Senior Databricks Engineer – Lead

This is a fully remote position, open to applicants in India.

📋 Description

• Create, develop, and sustain scalable data pipelines leveraging Azure Databricks

• Execute ETL/ELT processes utilizing PySpark, Spark SQL, and Python

• Enhance Spark job performance for efficiency, cost-effectiveness, and scalability

• Engage with structured and semi-structured data formats (Parquet, Delta, JSON, CSV)

• Construct and oversee Delta Lake tables (ACID compliance, time travel, schema evolution)

• Connect Databricks with Azure Data Lake Storage (ADLS Gen2)

• Formulate intricate queries and transformations using SQL

• Partner with data scientists, analysts, and stakeholders to facilitate analytics and ML applications

• Guarantee data quality, validation, and monitoring

• Adhere to best practices for security, access control, and governance in Azure

⛳️ Requirements

• Over 6 years of experience in Data Engineering

• Significant hands-on expertise with Azure Databricks

• Proficient in Python for data processing tasks

• Deep understanding of SQL (including joins, window functions, and performance tuning)

• Practical experience with Apache Spark / PySpark

• Familiarity with Delta Lake

• Knowledge of Azure Data Lake Storage (ADLS Gen2)

• Comprehension of distributed computing principles

• Experience utilizing Git for version control

• Hands-on experience with Azure Data Factory

• Familiarity with CI/CD pipelines (Azure DevOps, GitHub Actions)

• Basic knowledge of data modeling

• Awareness of cloud security and RBAC in Azure

• Exposure to streaming data technologies (Spark Structured Streaming, Event Hub, Kafka)

🏝️ Benefits

• Employees have the flexibility to work remotely