AI Quality Assurance – Evaluation Specialist

This is a fully remote position, open to applicants in Brazil, +5 more countries.

📋 Description

• Ensure that task instructions, source materials, reference solutions, and evaluation criteria are consistent and complete.

• Analyze AI agent execution traces, tool calls, and produced deliverables to determine the justification of outcomes.

• Examine grading logic to pinpoint fragile checks, erroneous expected answers, and unsupported rubric criteria.

• Identify instances where valid alternative solutions are unjustly penalized.

• Explore discrepancies between model performance, task defects, grader mistakes, and failures of the environment or tools.

• Conduct independent assessments of automated quality control findings instead of accepting them without validation.

• Document clear, evidence-supported findings along with actionable recommendations.

• Communicate uncertainty effectively and differentiate confirmed issues from possible problems.

• Confirm that task revisions address previously identified defects.

• Work collaboratively with relevant teams to enhance evaluation quality and reliability.

⛳️ Requirements

• Background in technical quality assurance, AI evaluation, data analysis, software testing, or a similar technical discipline.

• Proficient in reading and interpreting Python, SQL, shell scripts, structured data, and execution logs.

• Excellent analytical abilities, including the capacity to validate calculations and resolve conflicting evidence.

• Experience in evaluating the accuracy and completeness of technical or professional outputs.

• Strong written English skills with the ability to provide specific and reproducible feedback.

• Exceptional attention to detail when examining task requirements, grading behaviors, and model outputs.

• Capability to independently investigate issues and make decisions based on evidence.

🏝️ Benefits

• Opportunity to work in a dynamic and innovative environment.

• Access to professional development and training programs.

• Competitive salary and comprehensive benefits package.