Senior Data Engineer – Agents Systems

This is a fully remote position, open to applicants in United Kingdom.

📋 Description

• Take ownership of and enhance streaming data pipelines that facilitate live inference and real-time model serving within Kraken's AI infrastructure.

• Design and develop feature stores that provide low-latency, high-reliability features for production ML models.

• Implement and sustain streaming systems utilizing RisingWave, Apache Flink, or Kafka Streams, choosing the most suitable tool for each workload.

• Collaborate with ML engineers and AI infrastructure teams to establish data contracts, feature schemas, and pipeline service level agreements (SLAs).

• Lead the transition of pipelines toward real-time processing, reducing latency from hours to seconds where batch processes currently exist.

• Ensure data quality, observability, and auditability across all streaming and feature engineering systems.

• Contribute to the development of inference pipeline tools where data engineering and model serving converge.

• Assess new streaming and feature store technologies and help shape the team’s technical roadmap.

⛳️ Requirements

• A minimum of 5 years in data engineering, with at least 2 years dedicated to production streaming systems.

• Practical experience with RisingWave, Apache Flink, Kafka Streams, or similar stream processing frameworks.

• Strong grasp of feature store design principles, including online/offline consistency, point-in-time accuracy, and low-latency serving.

• Proven experience in building data pipelines that support production ML models or inference systems.

• Proficiency in Python and/or Scala is essential; SQL fluency is required.

• Familiarity with data quality frameworks, pipeline observability, and ownership of SLAs.

• Ability to thrive in a fast-paced, ambiguous environment within an AI-focused team.

🏝️ Benefits

• Our commitment to diversity and inclusion.

• Opportunities for professional development.

• Options for remote work.