Azure Databricks Developer

This is a fully remote position, open to applicants in United Kingdom.

📋 Description

• Design, develop, and enhance scalable data pipelines utilizing Azure Databricks, PySpark, Spark SQL, and Delta Lake.

• Construct reliable ETL/ELT solutions and data ingestion frameworks that accommodate both batch and streaming workloads.

• Integrate data from various sources including Azure SQL Database, Azure Data Lake Storage Gen2, Microsoft Fabric, APIs, SaaS platforms, and on-premise systems.

• Establish data quality, validation, monitoring, and governance measures throughout the data lifecycle.

• Assist in platform engineering efforts, such as CI/CD implementation, environment promotion, performance optimization, and operational support.

• Collaborate closely with Data Architects, Data Engineers, Data Analysts, and Solution Architects.

• Provide reliable, high-quality data products while adhering to security, governance, and operational standards.

⛳️ Requirements

• Proficiency in Azure Databricks, Apache Spark, PySpark, and Spark SQL.

• Experience in constructing enterprise-scale ETL/ELT pipelines within Azure settings.

• Strong development skills in Python and SQL.

• Familiarity with Azure Data Lake Storage Gen2 and Azure Data Factory or Microsoft Fabric Data Factory.

• Knowledge of Git version control and best practices in software engineering.

• Candidates must be able to obtain and maintain SC and NPPV3 clearance.

• Demonstrated experience in large-scale data processing and distributed computing technologies.

• Experience working in Agile delivery teams.

• Based in the UK.

• National mobility is required.

🏝️ Benefits

• 25 days of paid leave plus National Holidays.

• Pension Scheme with contributions matched up to 10%.

• Private Medical Coverage.

• Income Protection.

• Life Assurance.

• Flex Benefits Program.

• Remote working opportunities are available.

• Care leaver friendly employer.

• Reasonable adjustments to the application and assessment process.

• Disability Confident employer.