Research Scientist – Remote, US, LATAM

This is a fully remote position, open to applicants in Latin America.

📋 Description

• Conduct evaluation research by transforming public benchmarks and evaluation targets into innovative evaluation designs. Tackle complex issues such as construct validity, discrimination, headroom, and contamination.

• Develop benchmarks by creating evaluation packages in collaboration with subject-matter experts, ensuring each package includes expert-verified ground truth, comprehensive multi-model headroom results, and strict quality control (calibration layers, severity-weighted rubrics, deterministic verifiers).

• Manage expert recruitment, calibration, and review for a diverse pool specializing in coding, agentic/tool-use, and STEM/reasoning, serving as the final authority on correctness and frontier difficulty.

• Cultivate lab relationships by acting as a technical liaison for laboratories, supported by the CEO. Grasp their measurement objectives and translate them into effective evaluation designs.

• Oversee the delivery process by converting lab requests into successful sample packages, managing pilot projects from start to finish. Ensure that nothing is dispatched until it is ready for the lab.

⛳️ Requirements

• A research background in machine learning evaluation or benchmarking, including published/open benchmarks, evaluation research, or equivalent practical experience relied upon by labs.

• Extensive expertise in LLM benchmarking, particularly with a strong focus on code-model evaluation.

• Proficient understanding of how frontier models are assessed, including rubrics, pass rates, headroom, contamination, and the factors that differentiate tasks for a model.

• Demonstrated ability to maintain a rigorous standard for a team or expert pool.

• Proficient in English; proficiency in Spanish is a plus.

🏝️ Benefits

• Health insurance

• Professional development opportunities