Lead Databricks Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• Design, develop, and maintain scalable data pipelines utilizing Azure Databricks.

• Implement ETL/ELT workflows leveraging PySpark, Spark SQL, and Python.

• Optimize Spark jobs for enhanced performance, cost-effectiveness, and scalability.

• Work with both structured and semi-structured data formats (Parquet, Delta, JSON, CSV).

• Build and manage Delta Lake tables, incorporating ACID properties, time travel, and schema evolution.

• Integrate Databricks with Azure Data Lake Storage (ADLS Gen2).

• Develop complex queries and data transformations using SQL.

• Collaborate with data scientists, analysts, and stakeholders to facilitate analytics and machine learning applications.

• Ensure data quality, validation, and monitoring processes are adhered to.

• Follow best practices concerning security, access control, and governance in Azure.

⛳️ Requirements

• Over 6 years of experience in Data Engineering.

• Strong hands-on expertise with Azure Databricks.

• Proficient in Python for data processing tasks.

• In-depth knowledge of SQL, including joins, window functions, and performance optimization.

• Practical experience with Apache Spark / PySpark.

• Familiarity with Delta Lake.

• Knowledge of Azure Data Lake Storage (ADLS Gen2).

• Understanding of distributed computing principles.

• Experience using Git for version control.

• Experience with Azure Data Factory.

• Exposure to CI/CD pipelines, including Azure DevOps and GitHub Actions.

• Basic understanding of data modeling concepts.

• Familiarity with cloud security and role-based access control (RBAC) in Azure.

• Exposure to streaming data technologies, including Spark Structured Streaming, Event Hub, and Kafka.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Opportunities for professional development and continuous learning.

• Flexible work arrangements and a supportive work environment.

• Access to cutting-edge technology and tools.