Agent Quality, Evals Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop and sustain the MVP evaluation harness, including golden tasks, exception tasks, scorecard metrics, and regression packs.

• Integrate evaluations into the Continuous Integration (CI) pipeline to ensure that quality regressions cause builds and releases to fail.

• Collaborate with Product and the Tech Lead to define and uphold release-gate thresholds.

• Establish a foundation for future adversarial and drift-testing enhancements without unnecessarily expanding the MVP scope.

⛳️ Requirements

• Proven experience in evaluating machine learning (ML), large language models (LLM), or non-deterministic systems.

• Strong capability in test and benchmark design.

• Comfortable working with noisy metrics, thresholds, and probabilistic behavior.

• Proficient in scripting and automation techniques.

🏝️ Benefits

• Health insurance

• Retirement plans

• Paid time off

• Flexible work arrangements

• Opportunities for professional development