Azure Databricks Infrastructure Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Analyze dependencies specific to tenants, networking needs, DNS, Private Endpoints, and identity integrations.

• Establish migration waves and strategies for cutover.

• Transition Databricks workspaces across Azure tenants.

• Reconstruct or migrate workspaces, compute policies, job schedules, repositories, workflows, secret scopes, and access permissions.

• Move notebooks, libraries, packages, and configuration artifacts.

• Validate configurations for Unity Catalog and data governance.

• Implement infrastructure-as-code practices and automation.

• Migrate and validate Microsoft Entra ID integrations, managed identities, service principals, RBAC assignments, Databricks groups, and permissions.

• Oversee Key Vault migration and secret rotation tasks.

• Ensure adherence to the Principle of Least Privilege.

• Assist in security reviews and governance approvals.

• Evaluate and migrate related datasets and storage integrations.

• Validate Delta Lake tables, metadata, and schemas.

• Develop procedures for rollback and recovery.

• Automate migration and validation processes.

• Create a migration strategy and wave plan, detailed migration runbooks, cutover and rollback plans, and operational handover documentation.

• Lead migration efforts from assessment through production cutover with minimal oversight.

• Collaborate with architects, platform teams, security organizations, business stakeholders, and executive stakeholders.

⛳️ Requirements

• Candidates must have 13+ years of experience.

• A minimum of 7 years in Azure cloud is required.

• At least 5 years of experience in Azure Databricks administration and engineering is necessary.

• Proven experience in migrating Azure resources between tenants.

• Experience with large-scale data platform migrations is essential.

• Familiarity with designing secure cloud landing zones is required.

• Experience in implementing Infrastructure as Code is necessary.

• Experience in regulated enterprise environments is essential.

• Extensive knowledge in Databricks, Azure cloud services, identity and access management, networking, data engineering, and platform modernization is required.

• Required technical expertise includes Azure Databricks, Microsoft Entra ID, Azure Virtual Networks, Private Endpoints, Azure Key Vault, Azure Storage Accounts, Azure Data Lake Storage Gen2, Azure Monitor, Log Analytics, Azure Policy, Azure RBAC, Unity Catalog, Delta Lake, Databricks Repos, MLflow, cluster management, cluster policies, secret scopes, Databricks REST APIs, Terraform, Azure DevOps, GitHub Actions, PowerShell, Azure CLI, and Databricks CLI.

• Preferred certifications: Azure Solutions Architect Expert, Databricks Administrator, and Databricks Solution Architect.

• No OPT or fraudulent visas will be accepted.

• Must be available for telephonic and in-person interviews.

🏝️ Benefits

• Full-time employment.

• Long-term contract.

• Remote work opportunities.

• Equal opportunity employment.