Mid-Senior Databricks Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• Design and sustain scalable data pipelines utilizing Databricks, Python, and PySpark

• Construct and automate ETL workflows with Delta Live Tables (DLT)

• Enhance Spark jobs through techniques such as partitioning, shuffling, and performance tuning

• Apply Delta Lake optimization methods, including Z-Ordering and VACUUM

• Oversee data orchestration with Airflow or Azure Data Factory

• Ensure data quality, governance, and optimal pipeline performance

• Work in collaboration with cross-functional teams to provide effective data engineering solutions

⛳️ Requirements

• 3–6 years of experience in Data Engineering

• Proficient understanding of SQL, Python, and PySpark

• Practical experience with Databricks and Apache Spark

• Knowledge of Spark optimization, partitioning, and shuffling

• Familiarity with Delta Lake, Z-Ordering, and VACUUM

• Experience in creating ETL pipelines utilizing Delta Live Tables (DLT)

• Awareness of Apache Airflow or Azure Data Factory

• Understanding of data governance and data quality