
Head of Evaluation, Model Quality
Posted 1 day ago

Posted 1 day ago
This is a fully remote position, open to applicants in United States.
• Take ownership of the evaluation strategy, governance benchmarks, regression criteria, and the model-quality hiring plan.
• Identify the evidence necessary for release decisions and escalate any unsupported claims.
• Collaborate with causal research for validity and with engineering to ensure reliable evaluation execution.
• Develop the functional roadmap and suggest staffing and budget priorities in conjunction with the founding team.
• Compile model claims and customer use cases, mapping them to testable questions and suitable baselines.
• Create reproducible evaluation sets with documented provenance, held-out testing, leakage controls, and uncertainty estimates.
• Implement failure analysis across various populations, decision types, and distribution shifts.
• Produce release reviews that document trade-offs among quality, latency, cost, and known limitations.
• Monitor reproducibility, regression detection, consequential-use-case coverage, benchmark stability, cohort error, and verified-improvement timelines.
• Work closely with founders, bridging scientific and commercial boundaries.
• Engage in early evaluation work in a hands-on capacity.
• Demonstrated experience in leading evaluation, measurement, or model-quality programs.
• Proven track record of decisions being influenced by evaluation evidence.
• Strong skills in statistics and practical machine-learning evaluation.
• Proficient in building data and analysis pipelines hands-on.
• Capability to investigate confounding factors or leakage.
• Ability to evaluate the usefulness of metrics and articulate findings.
• Evidence of sound judgment, effective execution, and leadership.
• Must be eligible to work in the United States.
• Current position does not offer visa sponsorship.
• Currently, only equity-based compensation is available; no cash salary is offered at this time.
• Equity grant terms will be negotiated prior to joining.
• Any potential salary, incentive plans, benefits, hiring budget, or paid start date will depend on financing and a written agreement.
• Opportunity for remote work.
• Planned full-time role.
• Direct collaboration with founders.
• Working arrangements and travel expectations will be discussed.
Sedgwick
Cardinal Health
BlueCross BlueShield of Tennessee
Centene Corporation
Get handpicked remote jobs straight to your inbox weekly.