Senior Azure Databricks Architect

This is a fully remote position, open to applicants in Canada.

📋 Description

• Oversee the architecture and deployment of secure Azure Databricks environments tailored for enterprise clients

• Create secure and scalable Azure Databricks platform architectures

• Deploy and set up Azure Databricks workspaces utilizing Terraform and infrastructure-as-code methodologies

• Design and execute Azure Private Link and private endpoint connectivity for Databricks

• Configure private connectivity for front-end, back-end, and browser authentication

• Develop customer-managed VNet and VNet-injected Databricks deployments

• Implement Secure Cluster Connectivity and configurations with no public IP addresses

• Design private DNS zones, DNS resolution, routing, NSGs, user-defined routes, and firewall rules

• Integrate Databricks into hub-and-spoke and other enterprise Azure network frameworks

• Diagnose DNS, routing, authentication, and connectivity challenges alongside security and networking teams

• Create identity and access controls utilizing Microsoft Entra ID and Databricks Unity Catalog

• Set standards for architecture and deployment practices across development, testing, and production environments

• Establish CI/CD processes for Databricks infrastructure and platform configuration

• Generate architecture diagrams, design rationale, deployment documentation, and operational runbooks

• Offer technical leadership and insight to data engineers, cloud engineers, and client stakeholders

• Assist in implementation through production deployment

⛳️ Requirements

• Extensive experience in architecting and implementing Azure Databricks environments

• Practical experience deploying Azure Databricks with Terraform

• Proven track record of implementing Azure Private Link and private endpoints for Databricks

• Familiarity with VNet injection, customer-managed VNets, Secure Cluster Connectivity, and no-public-IP deployments

• In-depth understanding of Azure networking concepts, such as private DNS zones, DNS resolution, hub-and-spoke architectures, Azure firewalls, network security groups, user-defined routes, and ExpressRoute or VPN connectivity

• Experience with Terraform resources

• Background in designing secure connectivity between Databricks and Azure services like ADLS Gen2, Key Vault, Azure SQL, and Event Hubs

• Strong grasp of Databricks security, identity, access management, Unity Catalog, and service principals

• Experience in establishing CI/CD and infrastructure-as-code practices for enterprise data platforms

• Ability to convert security and business needs into actionable architectural decisions

• Excellent written and verbal communication skills, including experience engaging directly with clients and technical stakeholders

• Databricks architecture or engineering certification is preferred

• Relevant Microsoft Azure certification is preferred

• Experience with Databricks Asset Bundles is preferred

• Familiarity with Azure DevOps or GitHub Actions is preferred

• Knowledge of data governance, medallion architecture, observability, and cloud cost management is preferred

• Consulting or professional services experience is preferred

🏝️ Benefits

• Remote work opportunities available within Canada

• Engagement as a contractor