Databricks Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• Design, create, and maintain scalable data pipelines utilizing Azure Databricks

• Execute ETL/ELT workflows using PySpark, Spark SQL, and Python

• Enhance Spark job performance, cost efficiency, and scalability

• Manage structured and semi-structured data formats (Parquet, Delta, JSON, CSV)

• Construct and oversee Delta Lake tables, incorporating ACID properties, time travel, and schema evolution

• Integrate Databricks with Azure Data Lake Storage (ADLS Gen2)

• Create complex queries and transformations utilizing SQL

• Collaborate with data scientists, analysts, and stakeholders to facilitate analytics and machine learning initiatives

• Ensure data integrity, validation, and monitoring processes

• Adhere to best practices regarding security, access control, and governance within Azure

⛳️ Requirements

• A minimum of 4 years of experience in Data Engineering

• Extensive hands-on experience with Azure Databricks

• Proficient in Python for data processing tasks

• Strong understanding of SQL, including joins, window functions, and performance optimization

• Practical experience with Apache Spark / PySpark

• Familiarity with Delta Lake

• Knowledge of Azure Data Lake Storage (ADLS Gen2)

• Comprehension of distributed computing principles

• Experience with Git version control systems

• Background in Azure Data Factory

• Familiarity with CI/CD pipelines, including Azure DevOps and GitHub Actions

• Basic knowledge of data modeling techniques

• Understanding of cloud security and role-based access control (RBAC) in Azure

• Experience with streaming data technologies, including Spark Structured Streaming, Event Hub, and Kafka

🏝️ Benefits

• Competitive salary and performance-based bonuses

• Comprehensive health, dental, and vision insurance

• Opportunities for professional development and training

• Flexible work hours and remote work options

• Collaborative and innovative work environment