Senior Data Engineer

This is a fully remote position, open to applicants in United Kingdom.

📋 Description

• Develop and sustain ETL pipelines and coding within cloud data platforms.

• Assist with the ingestion process and the integration of new data sources.

• Collaborate with the Databricks Data Intelligence Platform to create data engineering workloads.

• Build raw, refined, and curated data layers while properly cataloguing assets.

• Ensure solutions are validated against both functional and non-functional requirements.

• Provide datasets, transformations, and performance-optimized data products.

• Enhance processes, engineering methodologies, and reusable tools.

• Monitor and assess pipeline performance while assisting in incident resolution.

• Confirm that documentation adheres to acceptance standards and is centrally approved.

• Participate actively in Agile ceremonies and governance discussions.

• Collaborate closely with architects, analysts, engineers, and broader delivery teams.

• Execute pipeline builds, data models, and integration patterns in line with the organization’s Target Operating Model for Data.

• Create scalable data flows and optimize data ingestion and transformation processes.

• Ensure that solutions comply with technical standards, performance criteria, and security protocols.

⛳️ Requirements

• Extensive experience with Python, PySpark, and SQL in the context of data engineering.

• Practical experience with Azure Databricks.

• In-depth knowledge of Azure Data Factory and Fabric.

• Familiarity with Microsoft Purview.

• Experience with event-driven data ingestion methods (Event Grid / Pub-Sub).

• Comprehension of SOLID principles, asynchronous programming, and Mediator/Factory design patterns.

• Experience in delivering unit and integration testing within Databricks.

• Understanding of secure ETL design incorporating Entra IMID/SCIM integration.

• Knowledge of Azure best practices, API Management (APIM), and platform governance.

• Capability to build and serve Power BI models using Databricks data sources.

• Previous experience in UK Public Sector environments is preferred.

• Candidates must be eligible to work with UK Government departments/BPSS.

• Strong communication skills for stakeholder engagement and cross-team collaboration.

• An analytical and solution-oriented mindset.

• Ability to work autonomously, take initiative, and drive progress.

• Commitment to clean, scalable, and well-documented engineering practices.

• Flexible, proactive, and comfortable in dynamic delivery settings.

🏝️ Benefits

• Technical autonomy, ownership, and challenging work.

• Remote work opportunities for candidates based in the UK.

• Contract or freelance engagement options.

• Immediate start available.