Databricks Administrator

This is a fully remote position, open to applicants in United States.

📋 Description

• Set up, maintain, and manage Azure Databricks workspaces and clusters across development, staging, and production environments.

• Monitor the health of clusters, performance metrics, resource utilization, and platform availability.

• Enhance configurations for clusters, SQL warehouses, and workspaces to improve performance and cost efficiency.

• Oversee workspace governance, including scheduling jobs, organizing notebooks, auditing logs, and upholding operational standards.

• Manage user access, workspace permissions, SQL warehouse configurations, and group-based security measures.

• Diagnose infrastructure issues, performance bottlenecks, connectivity problems, and workload failures.

• Assist in disaster recovery and high-availability planning, validation, and operational preparedness.

• Collaborate with data engineering teams to establish standards for Databricks usage, ensure reliability of workloads, troubleshoot platform issues, optimize Delta Lake performance, and encourage governed platform adoption.

• Implement and enforce security policies, RBAC, encryption standards, and access controls.

• Maintain documentation and runbooks detailing operational procedures.

• Aid in capacity planning, cost optimization, and initiatives for platform improvement.

• Uphold Azure Databricks security governance standards within a financial services context.

• Manage Unity Catalog permissions, metastores, catalogs, schemas, tables, volumes, and external locations.

• Handle workspace permissions, admin roles, service principals, groups, SCIM provisioning, and Microsoft Entra ID integration.

• Configure and review compute policies, cluster permissions, instance pool access, SQL warehouse permissions, and job ownership.

• Oversee secrets management, token usage, storage credential access, and configurations backed by Key Vault.

• Coordinate recurring access reviews, access certification, and remediation of excessive or outdated permissions.

• Analyze Databricks audit logs, security events, access changes, control findings, policy exceptions, and segregation-of-duties concerns.

• Facilitate audit and regulatory preparedness by maintaining evidence for access controls, configuration standards, policy enforcement, procedures, and change management.

• Work together with cybersecurity, compliance, risk, infrastructure, and data governance teams to ensure that Databricks controls align with financial services security, privacy, and compliance expectations.

⛳️ Requirements

• A minimum of 5 years of experience in administering cloud-based data platforms or analytics environments, with an emphasis on operational support, governance, and reliability.

• At least 3 years of hands-on experience managing Azure Databricks workspaces, Unity Catalog, clusters, jobs, SQL warehouses, and platform settings.

• Proficient in managing Databricks users, groups, workspace permissions, admin roles, service principals, SCIM provisioning, and Microsoft Entra ID integration.

• Experienced in administering Unity Catalog, including metastores, catalogs, schemas, tables, volumes, permissions, storage credentials, and external locations.

• Capable of configuring and maintaining compute policies, cluster permissions, instance pools, SQL warehouses, job ownership, and platform usage controls.

• Skilled in monitoring the health of the Databricks platform, cluster performance, job execution, workload failures, audit logs, and resource utilization.

• Strong troubleshooting abilities for Databricks platform issues, Spark workloads, connectivity challenges, permission errors, and performance bottlenecks.

• Proficient in Python, SQL, PowerShell, Databricks CLI, REST APIs, and automation practices.

• Experience in supporting operational controls, access reviews, audit evidence, change management, runbooks, and platform documentation.

• Knowledgeable in Databricks security governance, RBAC, least privilege principles, encryption, secrets management, audit readiness, and secure data access patterns.

• Understanding of Azure networking and security concepts, including private endpoints, firewalls, network security groups, storage access, Key Vault, and secure connectivity.

• Experience collaborating with data engineering teams to support reliable pipelines, troubleshoot workloads, optimize performance, establish platform standards, and ensure secure Databricks operations.

• Must possess valid U.S. work authorization that does not require employer-sponsored immigration assistance now or in the future.

• Databricks Platform Administrator certification or equivalent hands-on Azure Databricks administration experience is preferred.

• Previous experience supporting Databricks in financial services or other regulated environments is preferred.

• A background in database administration, cloud platform administration, or data platform operations is preferred.

🏝️ Benefits

• Medical/Rx/Dental/Vision coverage for employees and their eligible family members.

• Competitive PTO and vacation policies.

• One Friday off each month for Wellness Weekends.

• Company 401(k) plan with employer contributions after one year.

• Company-sponsored training and certification opportunities.

• Quarterly award ceremonies where top achievers are recognized and receive additional bonuses.

• Ongoing volunteer opportunities through the BHG Cares program.

• Emphasis on work/life balance.

• Opportunities for personal and professional development.

• Travel to attend the BHG Oars in Experience within the first 6 months and approximately 5% annually thereafter for team on-sites, meetings, and collaboration.