Senior Data Engineer

This is a fully remote position, open to applicants in United Kingdom.

📋 Description

• Take ownership of the design and execution of intricate components of the Microsoft Fabric data platform.

• Determine the onboarding and modeling processes for new data sources, including necessary controls and readiness for promotion.

• Spearhead the design of ingestion pipelines and warehouse models, overseeing everything from source review to live, reconciled, and well-documented production pipelines.

• Establish and refine engineering standards, naming conventions, load patterns, and requirements for re-runnability.

• Direct the design of the conformed warehouse layer and address discrepancies in identifiers, hierarchies, and status models from source systems.

• Conduct thorough peer reviews and serve as the technical approver for production modifications, particularly those involving regulatory pipelines and semantic models.

• Collaborate with the Product Owner to define, decompose, estimate, and prioritize data and business intelligence projects.

• Design validation, reconciliation, and monitoring controls, make judgments regarding materiality thresholds, and supply evidence to risk, compliance, and audit reviewers.

• Lead intricate incident diagnosis, including code and data rollback, root-cause analysis, and recovery efforts.

• Oversee semantic models to maintain business logic in version-controlled warehouse procedures and keep measures streamlined.

• Mentor engineers through reviews, pairing, and structured knowledge transfer initiatives.

• Potentially manage engineers, including setting objectives, development plans, conducting one-on-ones, and assessing performance.

• Engage with finance, risk, compliance, and business stakeholders to transform requirements into actionable solutions.

• Lead Agile planning, refine the backlog, facilitate retrospectives, and promote continuous improvement.

• Support data outputs for management information, financial reporting, self-service analytics, AI capabilities, and FCA/PRA regulatory submissions.

⛳️ Requirements

• Advanced skills in SQL development, optimization, and troubleshooting, particularly with complex stored procedure logic over bi-temporal, large data volumes.

• Proven experience in designing and delivering end-to-end pipelines or platforms within a cloud data platform, such as Microsoft Fabric, Azure Data Factory, Synapse, or Databricks.

• Comprehensive understanding of dimensional modeling utilizing the Kimball methodology, including conformed dimensions, slowly changing dimensions, surrogate key strategies, and incremental load patterns.

• Strong capabilities in semantic modeling and DAX, with a clear perspective on the division of logic between the warehouse and the model.

• Proficiency in Python for parsing, automation, and data management, preferably in a notebook setting.

• Solid knowledge of branching strategies, pull request policies, environment promotion, and secret management.

• Experience in operating change control within a regulated environment.

• Background in designing reconciliation, validation, and monitoring controls, along with establishing investigation thresholds.

• Ability to lead diagnosis and recovery efforts during live data incidents while communicating effectively.

• Genuine enthusiasm for mentoring and developing fellow engineers.

• Capability to manage priorities, expectations, and deliverables with senior stakeholders.

• Comfort in transforming vague requirements or undocumented source feeds into defined, actionable tasks.

• Hands-on experience with Microsoft Fabric Lakehouse, Fabric Data Warehouse, Data Pipelines, Notebooks, Git integration, and deployment pipelines is preferred.

• Experience in delivering data for regulatory reporting and collaborating with audit, risk, and compliance reviewers is advantageous.

• Familiarity with Wrap, SIPP, pensions, or investment platform data is beneficial.

• Experience with encrypted file transfers, key management, and automated decryption in a controlled environment is desirable.

• Understanding of curated data supporting AI use cases and associated governance is a plus.

• DP-700 certification is a welcome addition, though not mandatory.

🏝️ Benefits

• Non-contributory pension plan.

• Bonus opportunities.

• Enhanced parental leave policies.

• Paid time off available for emergencies.

• Health and wellness initiatives.

• Flexible working arrangements.

• A culture that prioritizes inclusion and diversity.

• Recognition and rewards for employee contributions.