Senior AI Product Manager – Evals & Annotations

This is a fully remote position, open to applicants in California, +1 more state.

📋 Description

• Take full ownership of Arize's Evals & Annotations product area, overseeing every aspect from evaluation creation and execution to human reviews, annotation workflows, and the overall evaluation lifecycle.

• Define the product strategy and roadmap by balancing customer requirements, platform investments, user experience enhancements, and potential opportunities in AI and agent development.

• Formulate strong opinions on workflows, create independent prototypes, and collaborate with Design to deliver intuitive and polished user experiences.

• Influence the future of evaluations for both autonomous and managed agents, including opportunities within the larger Dynatrace ecosystem.

• Partner with Engineering on the execution of evaluations, backend systems, orchestration, and data infrastructure.

• Engage in technical discussions regarding implementation methods and trade-offs.

• Conduct multiple customer discussions each week to gain insights into evaluation practices, identify challenges, and validate product direction.

• Work alongside cofounders and cross-functional teams to shape product direction, explore design possibilities, and prioritize initiatives.

• Make decisions on prioritization and guide products from defining problems to launching, adopting, and iterating.

⛳️ Requirements

• Extensive product management experience in independently managing technically complex developer-facing products; exact years may vary for candidates demonstrating exceptional technical expertise, product judgment, and ownership.

• Strong comprehension of LLM evaluations and AI development workflows, including automated evaluations, human feedback, experimentation, and iterative processes for AI applications and agents.

• Experience with developer tools, AI infrastructure, ML platforms, or related technical products.

• Solid knowledge of backend systems and software architecture.

• Strong technical acumen and hands-on engineering skills.

• Ability to independently troubleshoot issues and navigate code when necessary.

• Capability to implement small fixes or pull requests.

• Proficiency in building prototypes and creating technical proofs of concept.

• Skill in proposing technical implementations and assessing trade-offs.

• Capacity to engage in credible technical discussions with engineers.

• Excellent instincts in product and user experience; capability to define and prototype essential product experiences.

• Strong sense of ownership, initiative, and a proactive approach.

• Ability to navigate uncertainty and make informed trade-offs.

• Collaborative, low-ego demeanor in interactions with founders, engineers, and designers.

• Experience in working closely with technical customers.

• Strong communication skills with the ability to transition between strategic and execution-focused discussions.

🏝️ Benefits

• Competitive equity package.

• Medical insurance.

• Dental insurance.

• Vision insurance.

• 401(k) plan.

• Unlimited paid time off.

• Generous parental leave policy.

• Mental and wellness support.

• Monthly stipend for work-from-home co-working spaces.

• Optional in-person work opportunities in New York City and the San Francisco Bay Area.

• Regular discussions with industry experts, researchers, and ethicists.

• Culturally conscious events, such as LGBTQ trivia during Pride Month.

• Active Lady Arizers subgroup.