Cloud Engineer – Azure Databricks

This is a fully remote position, open to applicants in Mexico.

📋 Description

• Design, implement, and manage the Azure foundation that supports the client platform.

• Create and maintain reusable Terraform modules for provisioning Databricks workspaces, ADLS Gen2, Data Factory, Key Vault, networking, and Azure ML.

• Oversee remote state management, module versioning, drift detection, and deployment pipelines across development, testing, and production environments.

• Develop and implement private networking and security strategies, including private endpoints, hub-and-spoke topology, VNet injection, NSGs, and RBAC.

• Take ownership of ADLS Gen2 infrastructure, encompassing storage architecture, lifecycle policies, encryption, and access patterns.

• Manage operations on the Azure Data Factory platform, including integration runtime provisioning, linked-service credential design, and deployment automation.

• Provision and manage infrastructure for machine-learning workloads, covering Azure ML workspaces, GPU capacity planning, MLflow registry integration, and Unity Catalog connectivity.

• Establish and maintain monitoring using Azure Monitor and Log Analytics, including alerting and operational runbooks.

• Handle FinOps and capacity planning to optimize DBU and storage costs.

• Set platform standards, reference configurations, and multi-workspace strategies for the Databricks administration team.

• Provide guidance on Unity Catalog metastore design, permission models, and Delta Sharing configurations.

• Supervise identity and access management through Entra ID integration, SCIM provisioning, and identity federation.

• Enforce compute governance through cluster policies, instance pools, node-type standards, and serverless evaluation.

• Conduct cross-boundary diagnostics for failures affecting infrastructure, Databricks configuration, and workloads.

• Guide stakeholders and foster consensus regarding architectural approaches.

⛳️ Requirements

• A minimum of 6 years of experience in cloud infrastructure or platform engineering, with at least 4 years concentrating on Microsoft Azure.

• Expertise in Terraform with hands-on experience, including authoring production modules, managing remote state, and executing IaC via CI/CD.

• Proven hands-on experience in Databricks administration, which includes account and workspace management, Unity Catalog, cluster policies, identity federation, and cost governance.

• Extensive knowledge of Azure Networking & Security, involving private endpoints, VNets, hub-and-spoke design, NSGs, Entra ID, managed identities, RBAC, and Key Vault.

• Experience in enterprise-scale ADLS Gen2 design and access control.

• Familiarity with Azure Data Factory platform operations, including integration runtimes, managed VNet, credential management, and deployment automation.

• Strong skills in Python automation and scripting, along with PowerShell and/or Bash.

• Adequate knowledge of Spark and SQL to troubleshoot infrastructure and configuration-level performance issues.

• Proven track record as a senior technical partner who can guide stakeholders and build consensus around architectural strategies.

• Resilience, emotional intelligence, and a focus on agile delivery.

• In-depth understanding of the distinction between coding and engineering.

• Proficient in oral English.

• Proficient in Spanish.

• Desired: Experience with Databricks Asset Bundles, Terraform Databricks provider, and workspace-as-code patterns.

• Desired: Experience with Unity Catalog migration or multi-workspace consolidation.

• Desired: Knowledge of Azure Machine Learning, MLflow, or model-serving infrastructure.

• Desired: Familiarity with Kubernetes/AKS and containerized workloads.

• Desired: Experience with Policy as Code using Azure Policy, OPA, Sentinel, or Checkov.

• Desired: Familiarity with event-driven infrastructure using Event Hubs, Kafka, or Stream Analytics.

• Desired: Experience in formal FinOps practices.

• Desired: Experience with multi-region or multi-tenant Databricks deployments.

• Relevant certifications are preferred, including Databricks Certified Data Engineer Professional, Databricks platform administrator credential, Microsoft Azure Solutions Architect Expert (AZ-305), Azure Administrator Associate (AZ-104), DevOps Engineer Expert (AZ-400), or HashiCorp Certified Terraform Associate.

🏝️ Benefits

• Remote work flexibility.

• Adaptability to support your lifestyle.

• Global career opportunities.

• Engagement in high-impact projects.

• Collaboration within an international network of expertise.

• Participation in end-to-end digital transformation projects.

• Recognition as part of a workplace acknowledged as one of the most sustainable corporations worldwide.