Lead Test Engineer

This is a fully remote position, open to applicants in United Kingdom.

📋 Description

• Define and take ownership of the testing strategy for the Azure Databricks data platform.

• Design and implement a structured testing framework to replace the current ad-hoc pytest usage, establishing consistent patterns, standards, and reusable components for the team.

• Develop CI/CD pipelines (Azure DevOps) that execute automated tests and identify data and pipeline issues prior to reaching production.

• Introduce and integrate test-driven development (TDD) practices throughout the engineering team.

• Define and enforce data quality checks and validation rules during ingestion and transformation phases, utilizing tools such as Soda Data Quality or DQX.

• Lead, mentor, and provide direction for test engineers, reviewing their work to elevate testing standards across the platform.

• Collaborate with data engineers, architects, and platform leads to incorporate testing early in the development lifecycle.

• Produce comprehensive documentation of the testing strategy, framework, and standards for ongoing reference by the broader engineering team.

• Monitor and report on test coverage, defect trends, and platform quality metrics to program and technical leads.

⛳️ Requirements

• Demonstrated experience in defining and executing a testing strategy for a data platform, rather than merely executing pre-set test cases.

• Hands-on experience in developing testing frameworks for Databricks, including the creation and structuring of pytest-based tests.

• Practical experience with test-driven development (TDD) and its implementation within an engineering team.

• Experience in building or configuring CI/CD pipelines (Azure DevOps) to execute automated tests as part of the delivery process.

• Experience in leading or mentoring test engineers, with the capability to set standards and enhance quality practices across a team.

• Strong understanding of the challenges associated with data pipeline testing, including data quality, schema validation, and transformation accuracy.

• Ensure documentation is reviewed and approved by the central data platform team according to established acceptance criteria.

• Engage in communities of practice, ceremonies, and approval forums as necessary.

🏝️ Benefits

• Candidates must be eligible to work with UK Government departments / BPSS.