Senior Technical Product Manager – AI Agents, Evals & Reliability

This is a fully remote position, open to applicants in California.

📋 Description

• Conduct research to identify comprehensive end-to-end requirements for Artificial Intelligence (AI) systems, encompassing capabilities, behaviors, and user impacts.

• Convert model capabilities, data limitations, and evaluation outcomes into actionable product and system strategies.

• Make informed trade-offs concerning quality, latency, cost, reliability, and User Experience (UX).

• Collaborate closely with Machine Learning (ML), backend, and mobile engineers in system design, assessment, and iterative improvements.

• Define and refine evaluation frameworks utilizing offline metrics, online experiments, and human feedback.

• Propel project execution through well-defined specifications, sound judgment, and disciplined prioritization.

• Ensure prompt, safe, and reliable system deployments with robust feedback mechanisms.

• Take ownership of product quality throughout the entire process, focusing on correctness, predictability, and user trust.

• Align product strategy with AI capabilities, user requirements, and organizational priorities.

• Provide understandable, reliable, and trusted AI functionalities.

• Create clear roadmaps and priorities that allow for rapid iteration based on authentic user feedback.

• Coordinate teams to achieve AI product objectives with minimal friction.

⛳️ Requirements

• Experience in Artificial Intelligence (AI) and Machine Learning (ML).

• Strong foundation in computer science principles, including algorithms, data structures, and system architecture.

• Thorough understanding of Machine Learning (ML) principles and the operational behavior of modern Artificial Intelligence (AI) systems.

• Comfortable with reading, reviewing, and discussing technical design documentation.

• Hands-on experience with AI-driven products, particularly those based on Large Language Models (LLM).

• Background in model evaluation, prompt or pipeline iterations, and feedback mechanisms.

• Keen intuition regarding model limitations, hallucinations, biases, and drift.

• Extensive experience managing complex technical products from start to finish.

• Proven track record of collaborating effectively with senior engineers and Machine Learning (ML) teams.

• Strong judgment and decision-making skills in fast-paced, ambiguous environments.

• Ability to reconcile ambitious goals with technical and operational realities.

• Willingness to complete a 60-minute coding assessment.

🏝️ Benefits

• Medical insurance

• Dental insurance

• Vision insurance

• Savings Plan Options

• PTO (paid time off)

• Remote work / Work From Home