Program Manager, Human Evaluations – Recommendations

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee the complete execution of human evaluation and data operations projects from inception and planning to delivery.

• Collaborate with Product Management to convert recommendation strategies and roadmap priorities into evaluation frameworks, success metrics, and timelines.

• Manage human annotation service providers, including capacity planning, service level agreements (SLAs), quality performance, and issue escalations.

• Enhance operational efficiency across evaluation programs, focusing on cost per task, throughput, turnaround time, and speed to initial return on investment (IRR).

• Create clear and scalable guidelines and scoring rubrics to ensure consistent rater judgment.

• Supervise rater onboarding, calibration, and quality assurance workflows.

• Test and improve evaluation tasks to enhance clarity, inter-rater reliability, and feedback quality.

• Identify and address bottlenecks and obstacles proactively.

• Coordinate with product, machine learning, data science, and vendor partners to ensure aligned execution of evaluations at scale.

• Serve as a strategic partner to Product Management and contribute to establishing a shared vocabulary regarding recommendation quality.

⛳️ Requirements

• 4+ years of experience in human evaluations, data collection, labeling, or annotation operations within machine learning environments.

• Proven track record of implementing process enhancements or quality control systems for data collection requirements.

• Previous experience managing human annotation service providers, raters, or data labeling teams, including oversight of vendor performance and cost management.

• Demonstrated success in improving the operational efficiency of annotation programs, including throughput, turnaround, and cost metrics.

• Strong grasp of evaluation design principles, encompassing guidelines, rubrics, and scoring methodologies.

• Working knowledge of film and television content, including various genres, formats, and talent.

• Proven capability in the end-to-end management of complex, cross-functional programs.

• Strong program management skills with clear accountability for successful outcomes.

• Familiarity with human labeling platforms.

• Exceptional written and verbal communication abilities.

• Skill in synthesizing feedback into actionable recommendations and process enhancements.

• Awareness of rater subjectivity and biases in taste-based evaluations and design.

• Excellent organizational skills and a strong focus on execution.

• Experience with personalization, recommender systems, search, or ranking evaluations is a plus.

• Familiarity with offline human evaluations, online experimentation, and engagement metrics is advantageous.

• Experience evaluating at the list or page level is a bonus.

• Experience collaborating with Product Managers and Data Scientists on metric definition is a plus.

🏝️ Benefits

• Health Plans

• Mental Health support

• 401(k) Retirement Plan with employer match

• Stock Option Program

• Disability Programs

• Health Savings and Flexible Spending Accounts

• Family-forming benefits

• Life and Serious Injury Benefits

• Paid leave of absence programs

• Flexible time off for full-time salaried employees