Lead Data Engineer

This is a fully remote position, open to applicants in United Kingdom.

📋 Description

• Spearhead the design, development, and deployment of secure, scalable data pipelines and components for data platforms.

• Create and implement batch and real-time streaming solutions utilizing modern event-driven architectures.

• Take ownership of technical design choices across projects, ensuring a balance between scalability, security, performance, and delivery needs.

• Facilitate technical workshops, discovery sessions, and architecture discussions with client stakeholders.

• Evaluate existing client data platforms and devise practical strategies for modernization, migration, and integration.

• Establish and sustain data ingestion, transformation, and integration capabilities across various data sources.

• Develop cloud-native data platforms leveraging AWS services and open-source technologies.

• Guarantee that data quality, governance, lineage, and security standards are integrated throughout the data lifecycle.

• Promote engineering best practices, including code reviews, automated testing, CI/CD, and Infrastructure as Code.

• Assist multiple client engagements, aiding delivery teams in upholding consistent engineering quality and standards.

• Mentor and support engineers across project teams, offering technical guidance and leadership.

• Conduct data modeling, performance tuning, and root cause analysis to address complex data and platform challenges.

⛳️ Requirements

• Significant commercial experience in designing and implementing real-time streaming data platforms.

• Practical experience with technologies like Apache Kafka, AWS Kinesis, Apache Flink, Spark Streaming, Apache Pulsar, or similar event-driven technologies.

• Proven experience in building high-volume, low-latency, resilient data pipelines in production settings.

• Solid understanding of modern data engineering patterns, including batch, streaming, and hybrid architectures.

• Extensive experience in the development and operation of enterprise-scale data platforms.

• Strong SQL skills with a background in designing data models for both analytical and operational workloads.

• Proficiency in Python or other contemporary programming languages relevant to data engineering.

• Experience in configuring, tuning, and optimizing relational and NoSQL databases.

• Strong background working with AWS cloud services, including EC2, RDS, Lambda, Athena, Glue, S3, and IAM.

• Familiarity with Agile delivery environments.

• Robust understanding of Test-Driven Development (TDD) and automated testing methodologies.

• Experience in implementing CI/CD pipelines and engineering automation.

• Strong grasp of Infrastructure as Code principles.

• Experience with version control systems such as Git.

• Proven track record in delivering technology solutions within the UK Public Sector, Utilities, Energy, Water, Financial Services, or other highly regulated sectors.

• Understanding of security, governance, and compliance considerations in regulated environments.

• Experience with Terraform.

• Familiarity with Docker and Kubernetes.

• Knowledge of Ansible and Packer.

• Experience in Linux and Windows administration.

• Background in supporting AI, machine learning, or advanced analytics platforms.

• Experience with programs requiring UK Government security clearance.

• Strong technical leadership and problem-solving capabilities.

• Ability to communicate effectively with both technical and non-technical audiences.

• Experience in complex, regulated environments.

• A collaborative mindset and readiness to support and mentor others.

• A focus on providing practical solutions that deliver value to clients.

• A passion for creating reliable, scalable, and secure data platforms.

🏝️ Benefits

• 25 days of holiday plus bank holidays

• Company-paid medical insurance

• Life assurance

• Pension scheme

• Annual training allowance

• Wellbeing allowance

• Virtual GP

• Employee Assistance plan

• Additional perks