Database Administrator, Databricks

This is a fully remote position, open to applicants in United States.

📋 Description

• Enhance the data warehouse environment by transitioning the platform to Azure Databricks.

• Oversee and maintain Databricks workspaces across development, testing, and production settings.

• Configure and manage clusters, cluster policies, instance pools, SQL Warehouses, and schedule jobs.

• Monitor platform usage, performance, and capacity planning.

• Optimize cluster settings for cost efficiency, performance, and reliability.

• Manage workspace configurations, notebooks, libraries, and package dependencies.

• Experience in implementing Lakehouse architecture and Medallion data patterns (Bronze, Silver, Gold).

• Implement and sustain RBAC, Unity Catalog, workspace permissions, and data access controls.

• Collaborate with Azure platform teams to oversee Azure Key Vault, Azure Virtual Networks, Private Endpoints, Managed Identities, Azure Monitor, and Log Analytics.

• Troubleshoot issues related to cloud connectivity, networking, and authentication.

• Ensure secure connectivity between Databricks and enterprise systems.

• Provide support for Data Engineering, Analytics, AI/ML, and Data Science teams.

• Establish operational processes for environment provisioning, support, monitoring, and incident management.

• Perform and/or coordinate installations, upgrades, and patches for Microsoft SQL products and tools.

• Develop Infrastructure as Code (IaC) solutions utilizing Terraform and CI/CD pipelines.

• Automate workspace provisioning, user onboarding, cluster deployments, and policy management.

• Integrate Databricks deployments with Azure DevOps or GitHub Actions.

• Manage source control integration and release processes.

• Provide on-call support during off-hours for production, test, and development databases based on criticality.

⛳️ Requirements

• 5+ years of experience in managing enterprise data platforms.

• 3+ years of direct experience with Databricks administration.

• Strong background in: Azure Databricks, Unity Catalog, Delta Lake, Azure Data Lake Storage (ADLS), Azure Active Directory (Entra ID), and Azure Networking.

• Experience in implementing RBAC, security policies, and governance controls.

• Solid knowledge of monitoring, logging, troubleshooting, and performance tuning.

• Experience in managing production environments and platform support operations.

• Proficiency in Python, SQL, and scripting languages.

• Familiarity with Infrastructure as Code (Terraform preferred).

• Understanding of Microsoft Azure and the provisioning of SQL Server Databases.

• Knowledge of HIPAA, SOC 2, or other compliance frameworks pertinent to healthcare data.

• Bachelor’s degree from an accredited institution or equivalent experience.

🏝️ Benefits

• Health insurance

• 401(k)

• Paid time off

• Flexible work arrangements

• Opportunities for professional development