
AI Agent Evaluation Engineer
Posted Nov 12

Posted Nov 12
This is a fully remote position, open to applicants in Canada, +1 more country.
• Establish and uphold evaluation rubrics focused on adherence, outcomes, and conversation quality.
• Develop annotation guidelines and failure taxonomies; conduct calibration sessions and evaluate inter-rater agreement.
• Categorize and rank failure modes based on severity and frequency.
• Weekly review and annotate production conversations, incorporating multilingual evaluations.
• Identify failures' likely causes and direct evidence to the relevant stakeholders.
• Create and manage evaluation pipelines that integrate automated checks, LLM-based judges, and human reviews.
• Adjust automated judges to align with human labels.
• Formulate stratified sampling strategies.
• Establish and sustain golden sets and regression suites.
• Keep failure-mode coverage maps updated and address evaluation gaps.
• Design evaluations for critical edge cases and upstream failures.
• Set quality and latency standards.
• Provide reports on quality trends and uncertainties to stakeholders and clients.
• Work collaboratively with forward-deployed engineers, backend engineers, product managers, and voice/language engineering teams.
• Document methodologies to ensure reproducible evaluation practices.
• Minimum of 3 years of professional experience in software engineering, applied machine learning, or data-centric research roles.
• At least 1 to 2 years of hands-on experience with LLM-based systems in a production environment.
• Bachelor's or Master's degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or a related field, or equivalent practical experience.
• Experience in building and managing an end-to-end evaluation system for a production LLM product.
• In-depth knowledge of evaluation methodologies, such as golden sets, judges, regression suites, and calibration processes.
• Familiarity with LLM agents, including prompts, tool calls, context management, and multi-turn states.
• Proficient programming skills in Python and experience working with LLM APIs.
• Competence in writing rubrics and assessing inter-rater agreement.
• Ability to develop tests based on open-ended quality inquiries.
• Openness to listening to calls, labeling conversations, and constructing a failure taxonomy.
• Professional fluency in at least one language besides English.
• Excellent written communication skills.
• Completion of a self-paced CoderPad technical assessment may be necessary following resume evaluation.
• Comprehensive medical, dental, and vision plans to accommodate you and your family.
• Flexible PTO allowing you to take the time you need whenever necessary.
• Paid parental leave available for all new parents welcoming a child.
• Retirement savings plan to assist you in planning for your future.
• Budget for remote work setup to help you create an efficient home office.
• Monthly wellness and communication stipend to keep you connected and balanced.
• In-office meal program and commuter benefits provided for onsite employees.
• Equity as part of total compensation.
CVS Health Careers
Sourcegraph
MCA Connect
Lightly
Get handpicked remote jobs straight to your inbox weekly.