Senior Product Manager, AI Evaluation – Quality

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop evaluation strategies, methodologies, metrics, and acceptance criteria for Circana’s AI agents.

• Create evaluation frameworks for AI agent capabilities and agentic workflows.

• Collaborate with the Liquid AI Product Leader, engineering teams, and product verticals to assess agent performance prior to release.

• Generate representative test scenarios, golden datasets, benchmarks, and scoring methodologies, including LLM-as-a-judge techniques.

• Assess the quality, accuracy, consistency, reliability, safety, and client value of agents.

• Identify patterns of failure and convert insights into recommendations for product enhancements.

• Implement ongoing monitoring of production data to identify any performance changes post-launch.

• Document evaluation outcomes and share insights with both technical and business stakeholders.

• Advocate for uniform evaluation standards throughout Liquid AI product verticals.

⛳️ Requirements

• 4–6+ years of experience in product evaluation, data science, analytics, experimentation, quality assurance, or a related field.

• Proficiency in statistical evaluation, testing methodologies, and performance metrics.

• Experience evaluating AI agents or LLM-based systems, including LLM-as-a-judge scoring and analysis of production traces.

• Capability to design structured tests for complex or ambiguous scenarios.

• Strong analytical skills and meticulous attention to detail.

• Comfortable working in a fast-paced, early-stage environment with changing requirements.

• Ability to clearly communicate technical findings to a diverse audience.

• Bachelor’s degree in a quantitative, technical, or related field, or equivalent experience.

🏝️ Benefits

• Flexible and accommodating work environment.

• Paid time off.

• Medical, dental, and vision insurance.

• 401(k) plan.

• Eligibility for bonus pay.

• Support for reasonable accommodations for applicants with disabilities.

• Background checks conducted in accordance with local laws.

• AI-assisted resume screening.