Lead Data Engineer

This is a fully remote position, open to applicants in United Kingdom.

📋 Description

• Spearhead the design, development, and deployment of secure, scalable data pipelines and components for the data platform.

• Create and execute batch and real-time streaming solutions utilizing contemporary event-driven architectures.

• Take ownership of technical design decisions across various projects, ensuring a balance between scalability, security, performance, and delivery needs.

• Facilitate technical workshops, discovery sessions, and architecture discussions with client stakeholders.

• Evaluate client data platforms and formulate strategies for modernization, migration, and integration.

• Construct and uphold data ingestion, transformation, and integration capabilities.

• Develop cloud-native data platforms leveraging AWS services and open-source technologies.

• Maintain data quality, governance, lineage, and security throughout the data lifecycle.

• Promote engineering best practices, including code reviews, automated testing, CI/CD, and Infrastructure as Code.

• Support various client engagements while ensuring consistent engineering quality and standards.

• Mentor and assist engineers across project teams.

• Conduct data modeling, performance tuning, and root cause analysis to address complex data and platform challenges.

⛳️ Requirements

• Proven and hands-on Lead Data Engineer experience.

• Strong commercial expertise in designing and implementing real-time streaming data platforms.

• Practical experience with Apache Kafka, AWS Kinesis, Apache Flink, Spark Streaming, Apache Pulsar, or similar event-driven technologies.

• Experience in building high-volume, low-latency, resilient data pipelines in production settings.

• Solid understanding of batch, streaming, and hybrid architectures.

• Extensive experience in developing and managing enterprise-scale data platforms.

• Strong SQL skills and experience in designing data models.

• Proficiency in Python or other contemporary programming languages employed in data engineering.

• Experience in configuring, tuning, and optimizing relational and NoSQL databases.

• Significant experience with AWS cloud services, including EC2, RDS, Lambda, Athena, Glue, S3, and IAM.

• Familiarity with Agile delivery environments.

• Strong understanding of Test-Driven Development and automated testing methodologies.

• Experience in implementing CI/CD pipelines and engineering automation.

• Solid grasp of Infrastructure as Code principles.

• Experience with Git or similar version control systems.

• Proven track record of delivering technology solutions within the UK Public Sector, Utilities, Energy, Water, Financial Services, or other highly regulated sectors.

• Awareness of security, governance, and compliance considerations in regulated environments.

• Bachelor’s degree in Computer Science, Software Engineering, Data Engineering, Mathematics, or a related technical field, or equivalent practical experience.

• Familiarity with Terraform, Docker, Kubernetes, Ansible, Packer, Linux, Windows administration, AI, machine learning, or advanced analytics platforms, as well as UK Government security clearance is desirable.

🏝️ Benefits

• 25 days of holiday plus bank holidays.

• Company-paid medical insurance.

• Life assurance coverage.

• Pension scheme.

• Annual training allowance.

• Wellbeing allowance.

• Virtual GP services.

• Employee Assistance plan.