Tech Lead – Databricks, Data Platform

This is a fully remote position, open to applicants in India.

📋 Description

• Oversee the architecture, design, and execution of data platforms based on Databricks and cloud-native data pipelines.

• Create and manage CI/CD pipelines for data ingestion, transformation, dbt, SQL, notebooks, and Databricks workloads.

• Direct the comprehensive orchestration of Databricks Jobs and Workflows, encompassing ingestion, transformation, data quality checks, and dependencies.

• Establish and execute automated testing and quality gates within CI/CD pipelines.

• Promote Infrastructure as Code methodologies utilizing Terraform and reusable modules for Databricks and Azure infrastructure.

• Streamline Databricks platform operations, including workspace and cluster provisioning, runtime and library management, job deployment, configuration, and environment management.

• Develop and uphold IAM across Databricks and Azure, covering RBAC, TBAC, service principals, groups, roles, and access controls at the workspace, cluster, and table levels.

• Implement secure and scalable Unity Catalog and data governance frameworks in collaboration with architecture and security teams.

• Create reusable Terraform modules, Databricks job templates, Airflow DAG patterns, and engineering frameworks.

• Define and manage code promotion and release processes across development, QA, and production environments.

• Lead comprehensive orchestration using managed Airflow, ensuring effective scheduling, dependency management, monitoring, and recovery.

• Establish and track platform SLAs, SLOs, and SLIs; conduct incident triage, root cause analysis, and corrective measures.

• Apply FinOps best practices for monitoring, optimizing, and allocating costs associated with Databricks and cloud infrastructure.

• Assess and enhance platform tooling, architecture, reliability, security, performance, and developer productivity.

• Offer technical guidance and mentorship to engineers while establishing best practices, standards, and reusable patterns in engineering.

• Collaborate with Data Engineering, Analytics, Architecture, Security, and Infrastructure teams to provide enterprise data solutions.

• Contribute to cloud infrastructure, cybersecurity, disaster recovery, monitoring, logging, and operational readiness.

• Propel performance optimization and reliability enhancements for production data workloads.

⛳️ Requirements

• 10–12 years of experience in Data Engineering, Cloud Engineering, DevOps, Platform Engineering, SRE, or related technical fields.

• Strong hands-on experience with Databricks in production scenarios, including workspace and cluster management, Jobs/Workflows, Unity Catalog, and integration with orchestration platforms.

• Significant expertise with Microsoft Azure and cloud-native data platforms.

• Comprehensive knowledge of CI/CD and Git-based development workflows, utilizing tools such as Azure DevOps, GitHub Actions, GitLab CI, or similar.

• Extensive experience with Terraform / Infrastructure as Code and automation of cloud infrastructure.

• Practical experience with Apache Airflow or managed Airflow for orchestrating data pipelines.

• Strong programming/scripting capabilities in Python, Bash, or PowerShell.

• Experience in implementing automated testing, validation, and quality gates for data pipelines and models.

• Background in managing production workloads, including monitoring, logging, troubleshooting, performance tuning, and incident management.

• Strong grasp of cloud security, IAM, RBAC, data access controls, and governance.

• Experience in FinOps, cloud cost optimization, and monitoring resource utilization.

• Ability to provide technical leadership, mentor engineers, and advance engineering standards across teams.

• Excellent communication and stakeholder management skills, capable of effectively collaborating with both technical and non-technical stakeholders.

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline, or equivalent professional experience (preferred).

• Familiarity with dbt, SQL, Delta Lake, and contemporary data engineering practices (preferred).

• Experience with Unity Catalog and enterprise data governance (preferred).

• Background in designing reusable platform frameworks and accelerators (preferred).

• Experience in CPG, retail, manufacturing, or distribution environments (preferred).

• Knowledge of disaster recovery, business continuity, and highly available cloud architectures (preferred).

• Databricks or Azure certifications are advantageous.

🏝️ Benefits

• Remote work arrangement.