Senior Product Manager, AI Agents Testing

This is a fully remote position, open to applicants in Germany.

📋 Description

• Take ownership of the product strategy and roadmap for AI agent testing, including simulation, quality scoring, experimentation, regression detection, and conversation tracing.

• Deliver testing as a cohesive experience integrated within the builder and deployment process.

• Establish the end-to-end simulation process: generating scenarios based on actual conversation patterns, implementing automated pass/fail evaluations, and providing results that guide administrators to identify issues and their locations.

• Develop the experimentation framework — conducting A/B testing on agent behavior, implementing staged rollouts with statistical rigor, and ensuring safe iterations on tone and resolution strategies.

• Create a pre-publish readiness checkpoint that offers administrators a quantified assessment of risks prior to each deployment, including specific issues, coverage gaps, and comparisons to existing production behavior.

• Collaborate with machine learning, quality assurance, and platform teams regarding scoring methodologies, simulation infrastructure, and tracing architecture.

• Ensure that all tools are accessible to non-technical administrators — customer experience managers, bot developers, and operations leaders who require insights without needing to write code or submit engineering tickets.

⛳️ Requirements

• Several years of product management experience, including more than 2 years in developing for non-technical users within complex technical fields (such as QA tools, no-code platforms, admin consoles, and workflow builders) in B2B SaaS environments.

• Proven experience in delivering AI/ML products where evaluation and reliability were critical components, not mere afterthoughts.

• A clear understanding of the limitations of traditional testing methods for LLM-based systems, along with informed perspectives on effective alternatives.

• Capability to release platform features through user-facing product interfaces — you focus not only on building infrastructure but also on making it user-friendly.

• Experience in integrating acquired or related products into a seamless experience — merging functionalities from various teams, codebases, or organizations into a unified product.

• Demonstrated success in coordinating efforts across three or more engineering teams and multiple departments to create a cohesive product experience.

🏝️ Benefits

• Health insurance

• 401(k) matching

• Flexible work hours

• Paid time off

• Remote work options