Senior Product Manager

This is a fully remote position, open to applicants in Europe.

📋 Description

• Take ownership of the Evals & Quality charter, encompassing the problem, success metrics, and scope.

• Define and enhance core quality metrics such as first-pass-right, coverage, and regression detection.

• Manage rubrics and calibration for both automated judges and human raters.

• Design and implement the human quality-control system.

• Oversee the curation, coverage, and versioning of the golden dataset.

• Create reusable evaluation loops for other teams.

• Take charge of adevals.ai as a product, ensuring its design quality.

• Supply evidence for client RFP responses and creative enablement materials.

• Collaborate with Agent Architects on client-specific quality standards utilizing configurations and rubrics.

• Manage the complete product lifecycle from discovery to post-launch audit.

• Review every launch 14 days post-launch against its declared hypothesis.

⛳️ Requirements

• Proven experience taking ownership of quality, evaluations, or ML measurement as a product, complete with its own roadmap and user base.

• Experience in delivering measurement outcomes in a subjective domain.

• Direct involvement with golden datasets, rubric creation, judge calibration, and inter-rater reliability.

• Ability to derive product decisions from evidenced customer needs in a domain that required learning.

• Experience managing a metric that influenced decisions beyond your immediate team.

• Experience in presenting or defending methodologies to clients or buyers.

🏝️ Benefits

• 25 days of paid time off plus public holidays; we also operate a Flexible Time Off scheme.

• Health insurance and private medical coverage.

• Monthly stipend for wellness, fitness, and learning and development.

• Remote work opportunities from anywhere within your home country.

• Enhanced parental leave policies applicable to parents through birth, adoption, or surrogacy.

• Access to our Pencil office located in The Shard, London for our UK employees.

• Flexible working hours.