Head of Evaluation, Model Quality

Posted 1 day ago

This is a fully remote position, open to applicants in United States.

📋 Description

• Take ownership of the evaluation strategy, governance benchmarks, regression criteria, and the model-quality hiring plan.

• Identify the evidence necessary for release decisions and escalate any unsupported claims.

• Collaborate with causal research for validity and with engineering to ensure reliable evaluation execution.

• Develop the functional roadmap and suggest staffing and budget priorities in conjunction with the founding team.

• Compile model claims and customer use cases, mapping them to testable questions and suitable baselines.

• Create reproducible evaluation sets with documented provenance, held-out testing, leakage controls, and uncertainty estimates.

• Implement failure analysis across various populations, decision types, and distribution shifts.

• Produce release reviews that document trade-offs among quality, latency, cost, and known limitations.

• Monitor reproducibility, regression detection, consequential-use-case coverage, benchmark stability, cohort error, and verified-improvement timelines.

• Work closely with founders, bridging scientific and commercial boundaries.

• Engage in early evaluation work in a hands-on capacity.


⛳️ Requirements

• Demonstrated experience in leading evaluation, measurement, or model-quality programs.

• Proven track record of decisions being influenced by evaluation evidence.

• Strong skills in statistics and practical machine-learning evaluation.

• Proficient in building data and analysis pipelines hands-on.

• Capability to investigate confounding factors or leakage.

• Ability to evaluate the usefulness of metrics and articulate findings.

• Evidence of sound judgment, effective execution, and leadership.

• Must be eligible to work in the United States.

• Current position does not offer visa sponsorship.


🏝️ Benefits

• Currently, only equity-based compensation is available; no cash salary is offered at this time.

• Equity grant terms will be negotiated prior to joining.

• Any potential salary, incentive plans, benefits, hiring budget, or paid start date will depend on financing and a written agreement.

• Opportunity for remote work.

• Planned full-time role.

• Direct collaboration with founders.

• Working arrangements and travel expectations will be discussed.

People also viewed

Sedgwick12 hours ago

Construction Defect Adjuster

US flagAlabama OnlyFull-timeUncategorized
ApplyView job
Cardinal Health12 hours ago

Nurse, Individualized Care

US flagUnited States OnlyFull-timeUncategorized$68.5k – $107.6k/year
ApplyView job
BlueCross BlueShield of Tennessee12 hours ago

Health Navigator

US flagTennessee OnlyFull-timeUncategorized
ApplyView job
Centene Corporation12 hours ago

Care Navigator

US flagIllinois, +1 more stateFull-timeUncategorized$22 – $38/hour
ApplyView job
Centene Corporation12 hours ago

Clinical Review Nurse – Concurrent Review

US flagFlorida OnlyFull-timeUncategorized$27 – $48/hour
ApplyView job
Worldwide Clinical Trials12 hours ago

Clinician, Clinical Assessment Technologies

GB flagUnited Kingdom, +6 more countriesFull-timeUncategorized
ApplyView job

Never miss a great job!

Get handpicked remote jobs straight to your inbox weekly.

Trusted by 7,400+ designers