Staff Machine Learning Engineer

This is a fully remote position, open to applicants in California, +8 more states.

📋 Description

• Take ownership of the technical direction of the ML Platform — encompassing feature store, model hosting and serving, experimentation, and training infrastructure — while steering architectural decisions related to scalability, reliability, latency, and cost.

• Spearhead the design and execution of extensive platform initiatives from initial concept to production, collaborating across ML, data, and infrastructure teams.

• Identify and address systemic technical issues: ensuring online/offline feature parity, alleviating model deployment friction, enhancing experimentation velocity, optimizing GPU utilization, and managing cross-team dependencies.

• Establish and uphold a high standard of engineering quality through active code contributions, design reviews, and mentorship for platform and ML-adjacent engineers.

• Collaborate with ML engineering, data science, product, and platform leadership to convert ML strategy into actionable technical roadmaps.

• Define the standardized processes ML teams utilize to deploy models safely — covering everything from feature registration to canary rollout, monitoring, and rollback.

• Utilize AI-augmented development tools to enhance development speed and improve code quality.

⛳️ Requirements

• A minimum of 8 years of experience delivering intricate backend or infrastructure systems at scale.

• Hands-on experience in building or managing core ML infrastructure — including feature stores, model serving, experimentation platforms, training orchestration, or similar.

• Proficiency in a modern backend programming language such as Python, Java, Kotlin, Go, or Scala.

• In-depth understanding of distributed systems principles: consistency, latency, throughput, fault tolerance, and observability.

• Strong grasp of data modeling, query languages, and the online/offline data patterns that underpin ML systems.

• Proven technical leadership capabilities, with the ability to foster cross-team alignment and influence engineering, product, and business stakeholders.

• A Bachelor's degree in Computer Science or a related field, or equivalent practical experience.

🏝️ Benefits

• Competitive compensation and benefits programs.

• Flexibility to adapt to the changing needs of Toasters.

• Health insurance coverage.

• 401(k) matching.