Data Engineer – DataBricks

This is a fully remote position, open to applicants in India.

📋 Description

• Design and execute CI/CD pipelines for data and transformation projects, utilizing dbt on Databricks, SQL, and notebooks.

• Manage Databricks jobs and workflows from start to finish, encompassing ingestion, transformation, and quality assurance.

• Incorporate automated testing into CI/CD processes, including schema and contract validations for data models and tables.

• Apply FinOps best practices for monitoring costs and allocation throughout the EDP.

• Automate operations on the Databricks platform and associated services, such as workspace and cluster setup, library and runtime management, and job deployment/configuration.

• Establish and uphold identity and access management for Databricks and relevant cloud resources.

• Oversee access controls at the workspace, cluster, table, and view levels, including service principals, groups, roles, RBAC, and TBAC models.

• Provide Terraform modules, Airflow DAG patterns, and Databricks job templates to expedite project onboarding.

• Regularly assess and enhance tooling, pipelines, and platform architecture for improved reliability, security, and developer efficiency.

• Define the code promotion procedure to mitigate production impacts across domains.

• Manage comprehensive orchestration using managed Airflow.

• Assist in defining and monitoring platform SLA, SLO, and SLI metrics.

• Engage in incident response, including triage and root cause analysis.

⛳️ Requirements

• 4–7+ years of experience in DevOps, Cloud Engineering, Site Reliability Engineering, or Platform Engineering.

• A minimum of 2+ years supporting data/analytics platforms.

• Practical experience with Databricks in a production setting, including workspace and cluster management, job/workflow execution, and integration with orchestration tools.

• Robust experience with CI/CD pipelines, such as GitHub Actions, GitLab CI, Azure DevOps, or comparable tools.

• Proficiency in Git-based workflows.

• Extensive experience with Infrastructure as Code (IaC) and orchestration tools for the provisioning and management of Databricks and cloud infrastructure.

• Experience in implementing automated tests and quality gates within CI/CD pipelines.

• Background in managing production data workloads, focusing on monitoring, logging, performance optimization, and incident response.

• Proficient scripting skills in Python, Bash, or PowerShell for automation and integration purposes.

• Solid understanding of IT infrastructure technologies, cloud computing, cybersecurity, and disaster recovery practices.

• Familiarity with the Azure ecosystem, including the design, construction, and optimization of scalable data pipelines in cloud-native settings.

• Ability to collaborate effectively with data engineers, analytics engineers, architects, and security teams.

• Bachelor’s degree in Computer Science, Information Technology, or a related field is preferred, or equivalent professional experience.

• Experience in CPG, retail, manufacturing, or distribution sectors is preferred.

🏝️ Benefits

• Remote work arrangement