Senior Machine Learning Engineer, AI Safety Evaluations

This is a fully remote position, open to applicants in California.

📋 Description

• Design and uphold safety benchmarks for large language and multimodal models.

• Convert risk scenarios into test cases, datasets, and scoring rubrics.

• Establish metrics and protocols for evaluating safety in conjunction with model utility.

• Utilize baselines, uncertainty estimates, and failure analysis to differentiate significant improvements from noise.

• Create reusable Python pipelines featuring versioned datasets, model configurations, and experiment tracking.

• Develop automated red-teaming tools to identify failure modes and enhance evaluation coverage.

• Include human review to validate automated scoring and offer additional insights.

• Collaborate with researchers to investigate model failures and document actionable insights.

• Create regression tests to determine if model adjustments rectify identified issues.

• Partner with platform engineers to store and consolidate evaluation results.

• Convey findings, constraints, and emerging threats to researchers and decision-makers.

⛳️ Requirements

• Bachelor’s degree or equivalent experience.

• Over 6 years of experience in machine learning engineering or evaluation.

• Proven experience in developing benchmarks, metrics, and statistically robust comparisons for ML systems.

• Capability to translate vague evaluation inquiries into testable experiments.

• Proficient in Python.

• Experience in building maintainable shared tools, data pipelines, and experiment-tracking workflows.

• Ability to evaluate dataset quality and identify limitations in metrics and scoring methods.

• Competence in supporting conclusions with reproducible evidence.

• Strong communication skills to clearly articulate findings and trade-offs to researchers, engineering partners, and executives.

• Experience in evaluating LLMs, multimodal models, or agentic systems through adversarial testing or automated red-teaming.

• Contributions to public benchmarks or evaluation frameworks.

• A publication history and a desire to continue publishing in top-tier venues.

• Experience in validating model-based evaluators against human judgments or designing human-in-the-loop evaluation workflows.

• Strong model evaluation capabilities and dedication to sound statistical practices, including confidence intervals.

🏝️ Benefits

• Equity.

• Benefits.