Technical AI Evaluation Analyst – LATAM

This is a fully remote position, open to applicants in Argentina, +5 more countries.

📋 Description

• Ensure task quality by verifying instructions, source materials, reference solutions, and evaluation criteria for consistency and thoroughness.

• Analyze AI agent execution traces, tool interactions, and deliverables to assess the justification of outcomes.

• Review grading logic to pinpoint fragile checks, incorrect expected answers, unsupported rubric criteria, and unjust penalties for valid alternative solutions.

• Examine discrepancies between model performance, grader outputs, and anticipated results.

• Differentiate between authentic model limitations and issues arising from task defects, grader inaccuracies, or failures in the environment or tools.

• Conduct independent evaluations of automated quality control findings instead of accepting them at face value.

• Compile clear, evidence-based findings and offer actionable, reproducible feedback.

• Identify areas of uncertainty and ensure that implemented changes effectively address previously noted issues.

⛳️ Requirements

• Proficient in reading and interpreting Python, SQL, shell scripts, structured data, and execution logs.

• Background in reviewing technical workflows, software behavior, data outputs, or evaluation logic.

• Strong analytical capabilities, including the ability to verify calculations and reconcile contradictory evidence.

• Proven ability to evaluate the accuracy and completeness of technical deliverables.

• Excellent written communication skills in English, with experience in providing clear, specific, and reproducible feedback.

• Exceptional attention to detail when detecting inconsistencies, missing information, and evaluation flaws.

🏝️ Benefits

• Opportunity to work in a dynamic and innovative environment.

• Competitive salary and performance-based incentives.

• Comprehensive health benefits and wellness programs.

• Professional development and growth opportunities.

• Flexible work hours and remote work options.