Senior Databricks Engineer – Lead

This is a fully remote position, open to applicants in India.

📋 Description

• Design, develop, and sustain scalable data pipelines utilizing Azure Databricks.

• Execute ETL/ELT workflows leveraging PySpark, Spark SQL, and Python.

• Enhance Spark jobs for improved performance, cost efficiency, and scalability.

• Work with both structured and semi-structured data formats such as Parquet, Delta, JSON, and CSV.

• Construct and manage Delta Lake tables with features like ACID transactions, time travel, and schema evolution.

• Integrate Databricks with Azure Data Lake Storage Gen2.

• Create complex queries and transformations using SQL.

• Collaborate with data scientists, analysts, and stakeholders to facilitate analytics and machine learning use cases.

• Ensure data quality through validation and monitoring processes.

• Adhere to best practices regarding security, access control, and governance within Azure.

⛳️ Requirements

• Over 6 years of experience in Data Engineering.

• Strong practical experience with Azure Databricks.

• Proficiency in Python for data processing tasks.

• Solid understanding of SQL, including joins, window functions, and performance optimization techniques.

• Hands-on experience with Apache Spark and PySpark.

• Familiarity with Delta Lake.

• Knowledge of Azure Data Lake Storage Gen2.

• Comprehension of distributed computing concepts.

• Experience utilizing Git for version control.

• Proficiency with Azure Data Factory.

• Exposure to CI/CD pipelines, including Azure DevOps and GitHub Actions.

• Basic understanding of data modeling principles.

• Awareness of cloud security and Role-Based Access Control (RBAC) in Azure.

• Familiarity with streaming data technologies such as Spark Structured Streaming, Event Hub, and Kafka.

🏝️ Benefits

• Remote work is permitted.