Senior Data Engineer - Databricks

This is a fully remote position, open to applicants in India.

📋 Description

• Design and implement Azure Databricks in a cloud environment utilizing Azure Cloud services.

• Execute ETL design, development, and deployment to the Cloud Service.

• Collaborate with Onshore teams to understand their business objectives and contribute to the successful delivery of workstreams.

• Create and enhance model codes for improved execution speed.

⛳️ Requirements

• 3 to 7 years of experience in developing data ingestion, processing, and analytical pipelines for big data, relational databases, NoSQL, and data warehouse solutions.

• Extensive practical experience in implementing data migration and processing using Azure services such as Databricks, ADLS, Azure Data Factory, Azure Functions, Synapse/DW, Azure SQL DB, etc.

• Proficient in Azure Databricks and Pyspark for developing ETL pipelines.

• Utilize Pyspark within Databricks notebooks for data processing and transformations.

• Integrate and orchestrate data from diverse sources while ensuring data quality and accuracy.

• Develop reusable rules for Data Quality and necessary transformations.

• Optimize data pipelines for enhanced performance and scalability.

• Implement data governance and security measures to adhere to regulations.

• Maintain documentation of data pipelines and workflows for knowledge sharing.

• Proficiency in Databricks, Azure Data Factory, Python, and SQL is essential.

• Familiarity with ETL principles, data warehousing, and big data technologies.

• Strong problem-solving abilities and excellent communication skills.

🏝️ Benefits

• Support, coaching, and feedback from some of the most engaging colleagues around.

• Opportunities to develop new skills and advance your career.

• The freedom and flexibility to manage your role in a way that suits you best.