Senior Product Manager, Observability Tools

This is a fully remote position, open to applicants in Argentina.

📋 Description

• Take ownership of scenario development through the simulation agent, utilizing tools for the creation and curation of realistic test scenarios.

• Manage evaluation surfaces, monitor errors, and conduct structured data assessments.

• Establish techniques for identifying, highlighting, and classifying mistakes made by Generative Agents on a large scale.

• Oversee the Previewer to evaluate configuration adjustments in simulated conversations.

• Direct the Conversation Explorer for analyzing complete production dialogues, encompassing agent thoughts and actions.

• Advocate for simulation-driven quality in collaboration with Sales, Customer Success, and Research.

• Collaborate with Research and Engineering on agent architecture, reasoning paths, tool interactions, and scenario creation.

• Manage the trust component of the review process by addressing desynchronization issues, incomplete traces, and ambiguous mistake classifications.

• Supervise the data layer, which includes models, dashboards, feeds, and APIs, ensuring ASAPP’s platform is reliable and actionable.

⛳️ Requirements

• Over 5 years of experience in product management, with significant involvement in testing, quality, or review/analytics products within the B2B SaaS sector.

• Hands-on experience in developing products related to LLMs, AI agents, or other applied AI technologies.

• Capability to analyze agent behavior, design prompts and scenarios, or evaluate model outputs.

• Proven experience in delivering tools that simulate, test, or preview system functionalities prior to production.

• Background in creating review or audit tools for complex and unstructured content.

• Skill in translating ambiguous quality indicators into clear and defensible classifications.

• Comfort in collaborating with research and engineering teams on reasoning traces, tool usage logs, and evaluation strategies.

• Demonstrated history of conducting discovery sessions with internal or external users and converting insights into structured requirements.

• Exceptional written and verbal communication skills.

• Familiarity with contact centers, conversational AI, or customer experience analytics is advantageous.

• Knowledge of evaluation frameworks or methodologies for generative AI systems.

• Previous experience in a startup or rapidly scaling environment.

🏝️ Benefits

• Competitive salary

• Stock options

• Family healthcare coverage

• Mac equipment provided

• USD 150 monthly in flexible benefits

• 3 weeks of vacation

• English language lessons