Azure Databricks Administrator

This is a fully remote position, open to applicants in United States.

📋 Description

• Set up, manage, and oversee Azure Databricks workspaces and clusters across development, staging, and production settings.

• Monitor the health of clusters, performance metrics, resource usage, and platform availability.

• Enhance cluster, SQL warehouse, and workspace configurations for optimal performance and cost effectiveness.

• Oversee workspace governance, which includes job scheduling, notebook organization, audit logging, and operational standards.

• Manage user access, workspace permissions, SQL warehouse configurations, and group-based security measures.

• Diagnose infrastructure issues, performance bottlenecks, connectivity challenges, and workload failures.

• Assist with disaster recovery and high-availability planning, validation, and operational readiness.

• Collaborate with data engineering, analytics, cybersecurity, compliance, and business teams to ensure platform reliability, governance, security, and support.

• Enhance Delta Lake performance and advocate for governed platform adoption.

• Implement and enforce security policies, RBAC, encryption standards, and access controls.

• Maintain documentation and runbooks outlining operational procedures.

• Assist with capacity planning, cost optimization, and initiatives aimed at platform enhancement.

• Manage Unity Catalog permissions, metastore access, catalogs, schemas, tables, volumes, and external locations.

• Oversee workspace permissions, admin roles, service principals, groups, SCIM provisioning, and Microsoft Entra ID integration.

• Configure and evaluate compute policies, cluster permissions, instance pool access, SQL warehouse permissions, and job ownership.

• Manage secrets, token utilization, storage credential access, and Key Vault-backed setups.

• Coordinate regular access reviews, access certification, and addressing excessive, inappropriate, or outdated permissions.

• Analyze Databricks audit logs, security incidents, access modifications, and control findings.

• Support audit and regulatory compliance by maintaining documentation for access controls, configuration standards, policy enforcement, operational procedures, and change management.

• Collaborate with cybersecurity, compliance, risk, infrastructure, and data governance teams to align controls with financial services security, privacy, and compliance requirements.

⛳️ Requirements

• Over 5 years of experience managing cloud-based data platforms or analytics environments, with a focus on operational support, governance, and reliability.

• More than 3 years of hands-on experience with Azure Databricks workspaces, Unity Catalog, clusters, jobs, SQL warehouses, and platform configurations.

• Strong expertise in managing Databricks users, groups, workspace permissions, admin roles, service principals, SCIM provisioning, and Microsoft Entra ID integration.

• Experience overseeing Unity Catalog, including metastores, catalogs, schemas, tables, volumes, permissions, storage credentials, and external locations.

• Proficient in configuring and maintaining compute policies, cluster permissions, instance pools, SQL warehouses, job ownership, and platform usage controls.

• Experience in monitoring the health of the Databricks platform, cluster performance, job execution, workload failures, audit logs, and resource utilization.

• Strong troubleshooting capabilities for issues within the Databricks platform, Spark workloads, connectivity problems, permission errors, and performance bottlenecks.

• Familiarity with Python, SQL, PowerShell, Databricks CLI, REST APIs, and automation practices.

• Experience in supporting operational controls, access reviews, audit evidence, change management, runbooks, and platform documentation.

• Knowledge of Databricks security governance, RBAC, least privilege principles, encryption, secrets management, audit readiness, and secure data access methodologies.

• Understanding of Azure networking and security concepts, including private endpoints, firewalls, network security groups, storage access, Key Vault, and secure connectivity.

• Experience collaborating with data engineering teams to ensure reliable pipelines, troubleshoot workloads, tune performance, establish platform standards, and secure Databricks operations.

• Valid U.S. work authorization that does not require employer-provided immigration sponsorship now or in the future.

• All positions require travel during the initial 6 months of employment to attend the BHG Oars in Experience.

• Preferred: Databricks Platform Administrator certification or equivalent practical experience in Azure Databricks administration.

• Preferred: Experience supporting Databricks within financial services or another regulated sector.

• Preferred: Experience in creating runbooks, operational standards, monitoring procedures, access review processes, and documentation for administrators.

• Preferred: Background in database administration, cloud platform administration, or data platform operations supporting enterprise analytics workloads.

🏝️ Benefits

• Contract until the end of March (with a firm end date).

• Remote work opportunity.

• Approximately 5% annual travel after the first 6 months for occasional team on-site meetings, team gatherings, and cross-functional collaboration.