Senior Data Engineer – Agents Systems

This is a fully remote position, open to applicants in United States.

📋 Description

• Take ownership and enhance streaming data pipelines that facilitate live inference and real-time model deployment within Kraken's AI ecosystem.

• Design and create feature stores that deliver low-latency, high-reliability features to production machine learning models.

• Implement and sustain streaming systems utilizing RisingWave, Apache Flink, or Kafka Streams, selecting the most appropriate tool based on workload requirements.

• Collaborate with machine learning engineers and AI infrastructure teams to establish data contracts, feature schemas, and pipeline service level agreements (SLAs).

• Propel pipelines towards real-time operation, transitioning from batch processing that currently takes hours to processing that takes mere seconds.

• Ensure data quality, observability, and auditability across all streaming and feature engineering systems.

• Contribute to tooling for inference pipelines where data engineering intersects with model serving.

• Assess emerging technologies in streaming and feature stores, influencing the team's technical roadmap.

⛳️ Requirements

• A minimum of 5 years in data engineering, with at least 2 years dedicated to production-focused streaming systems.

• Practical experience with RisingWave, Apache Flink, Kafka Streams, or similar stream processing frameworks.

• In-depth knowledge of feature store architecture, including online/offline consistency, point-in-time accuracy, and low-latency serving.

• Proven experience in building data pipelines that support production machine learning models or inference systems.

• Proficient in Python and/or Scala; SQL fluency is essential.

• Familiarity with data quality frameworks, pipeline observability, and ownership of SLAs.

• Ability to thrive in a fast-paced, ambiguous environment as part of an AI-centric team.

🏝️ Benefits

• Bonus program

• Equity program

• Wellness allowance

• Medical insurance

• Dental insurance

• Vision insurance

• 401(k) plan