AI Agent Evaluation Engineer

Posted Nov 12

This is a fully remote position, open to applicants in Canada, +1 more country.

📋 Description

• Establish and uphold evaluation rubrics focused on adherence, outcomes, and conversation quality.

• Develop annotation guidelines and failure taxonomies; conduct calibration sessions and evaluate inter-rater agreement.

• Categorize and rank failure modes based on severity and frequency.

• Weekly review and annotate production conversations, incorporating multilingual evaluations.

• Identify failures' likely causes and direct evidence to the relevant stakeholders.

• Create and manage evaluation pipelines that integrate automated checks, LLM-based judges, and human reviews.

• Adjust automated judges to align with human labels.

• Formulate stratified sampling strategies.

• Establish and sustain golden sets and regression suites.

• Keep failure-mode coverage maps updated and address evaluation gaps.

• Design evaluations for critical edge cases and upstream failures.

• Set quality and latency standards.

• Provide reports on quality trends and uncertainties to stakeholders and clients.

• Work collaboratively with forward-deployed engineers, backend engineers, product managers, and voice/language engineering teams.

• Document methodologies to ensure reproducible evaluation practices.


⛳️ Requirements

• Minimum of 3 years of professional experience in software engineering, applied machine learning, or data-centric research roles.

• At least 1 to 2 years of hands-on experience with LLM-based systems in a production environment.

• Bachelor's or Master's degree in Computer Science, Computational Linguistics, Cognitive Science, Statistics, or a related field, or equivalent practical experience.

• Experience in building and managing an end-to-end evaluation system for a production LLM product.

• In-depth knowledge of evaluation methodologies, such as golden sets, judges, regression suites, and calibration processes.

• Familiarity with LLM agents, including prompts, tool calls, context management, and multi-turn states.

• Proficient programming skills in Python and experience working with LLM APIs.

• Competence in writing rubrics and assessing inter-rater agreement.

• Ability to develop tests based on open-ended quality inquiries.

• Openness to listening to calls, labeling conversations, and constructing a failure taxonomy.

• Professional fluency in at least one language besides English.

• Excellent written communication skills.

• Completion of a self-paced CoderPad technical assessment may be necessary following resume evaluation.


🏝️ Benefits

• Comprehensive medical, dental, and vision plans to accommodate you and your family.

• Flexible PTO allowing you to take the time you need whenever necessary.

• Paid parental leave available for all new parents welcoming a child.

• Retirement savings plan to assist you in planning for your future.

• Budget for remote work setup to help you create an efficient home office.

• Monthly wellness and communication stipend to keep you connected and balanced.

• In-office meal program and commuter benefits provided for onsite employees.

• Equity as part of total compensation.

People also viewed

CVS Health Careers6 hours ago

Senior Analyst, Clinical AI and Data Strategy

US flagTexas OnlyFull-timeArtificial Intelligence$47k – $102k/year
ApplyView job
Sourcegraph6 hours ago

AI Transformation Architect

US flagUnited States OnlyFull-timeArtificial Intelligence$200k/year
ApplyView job
MCA Connect6 hours ago

Senior Consultant, Data & AI – Microsoft

US flagUnited States OnlyFull-timeArtificial Intelligence$120k – $145k/year
ApplyView job
Lightly7 hours ago

AI Operations Monitor – Digital Operations & Incident Response

CH flagSwitzerland OnlyFreelanceArtificial Intelligence$20 – $30/hour
ApplyView job
WRS Health7 hours ago

AI & Automation Marketing Lead

US flagUnited States OnlyFreelanceArtificial Intelligence
ApplyView job
WRS Health7 hours ago

AI and Automation Marketing Lead

US flagUnited States OnlyFreelanceArtificial Intelligence
ApplyView job

Never miss a great job!

Get handpicked remote jobs straight to your inbox weekly.

Trusted by 7,400+ designers