Staff AI Evaluation Lead

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee intricate, high-impact automation and evaluation projects across various workflows or teams.

• Design comprehensive workflows that integrate datasets, evaluations, automation processes, and human-in-the-loop elements.

• Develop reusable harnesses, components, and evaluation pipelines that operate in production without requiring manual intervention.

• Establish dataset standards, evaluation methodologies, failure taxonomies, and quality measurements throughout AI Operations.

• Ensure that projects adhere to set standards and take ownership of the creation and adoption of these standards.

• Create, sustain, validate, and update organizational metric definitions.

• Set production quality benchmarks and identify points where human review is necessary.

• Determine evaluation depth based on system risk and transparently present accepted trade-offs to stakeholders.

• Specify re-baselining requirements for changes in models and platforms.

• Assess new models and AI capabilities to make decisions on organizational adoption.

• Lead the reengineering of workflows, tools, and processes to enhance performance and resilience within AI Operations.

• Collaborate with Engineering, Product, and AI teams to provide solutions and synchronize priorities.

• Establish success metrics and relay performance insights and recommendations to leadership.

• Mentor team members and enhance the team’s automation and evaluation expertise.

• Keep abreast of emerging tools and methodologies, pilot them, and convert them into actionable improvements.

⛳️ Requirements

• 8+ years of professional experience in AI, machine learning, operational automation, or a related domain.

• Demonstrated capability to spearhead complex automation or evaluation initiatives across systems or teams.

• Experience in designing, constructing, and scaling evaluation frameworks, datasets, and quality systems at scale for LLMs or AI products.

• Proficiency in SQL, JSON, APIs, and scripting with AI support.

• Responsibility for the technical direction of the evaluation stack.

• Extensive knowledge of LLM and agentic system behavior, automation platforms, and system design, as shown in built systems.

• Experience with data pipelines, APIs, and production systems.

• Proven ability to influence cross-functional stakeholders and align priorities effectively.

• Established track record of defining metrics and driving measurable business results.

• Experience in mentoring or leading technical contributors is preferred.

• Experience evaluating agentic systems in large-scale production is preferred.

• Experience in setting technical standards adopted organization-wide is preferred.

🏝️ Benefits

• Equity for full-time employees.

• 401(k) matching dollar for dollar, up to 4% of eligible pay.

• Comprehensive medical, dental, and vision plans, including coverage for pets.

• $10,000 per year in education reimbursement.

• Participation in Employee Resource Groups and the Inclusion Council.

• Flexible paid time off (PTO).

• 9 paid holidays.

• A week-long break at year-end.

• 14 weeks of fully paid parental leave.

• 6 additional weeks of paid childbirth recovery leave for birthing parents.

• Flexible return-to-work schedule.

• One-time $2,000 Calvin Care Cash incentive for employees welcoming new family members.

• Flexible work environment.

• Hardware and software provided for remote work.