Research Engineer / Scientist, Data & Evals

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop synthetic data pipelines that generate, filter, and curate training and evaluation samples for intricate legal reasoning and agentic workflows.

• Create evaluations and verifiers that mirror actual legal work, calibrate them against expert opinions, and analyze any failures.

• Establish and maintain capability benchmarks, including the Legal Agent Benchmark (LAB).

• Construct product-driven evaluations that monitor model performance in real customer workflows.

• Collaborate with legal domain specialists to define quality metrics, rubrics, and grading systems.

• Take ownership of the quality of training and evaluation data, focusing on aspects such as provenance, coverage, and contamination safeguards.

• Link product usage and customer feedback with new evaluations and training data.

• Collaborate with researchers to analyze model failures, differentiate genuine progress from misleading metrics, and inform training and release strategies.

⛳️ Requirements

• Proven experience in building data pipelines, ML evaluation systems, or experimentation tools.

• Proficient in Python programming.

• Experience in managing production-quality code.

• A rigorous approach to measurement, incorporating statistical reasoning and experimental design.

• Awareness of situations where metrics may be misleading.

• Capability to convert ambiguous objectives or expert insights into structured datasets, rubrics, and measurable criteria.

• Familiarity with synthetic data, LLM evaluations, automated raters, or preference data is advantageous.

• Strong experience in data and measurement engineering closely related to ML model development is beneficial.

• Positions available at various levels.

🏝️ Benefits

• Equity plan.

• Comprehensive health coverage.

• Dental coverage.

• Vision coverage.

• Retirement benefits with a 401(k) match up to 4%.

• Flexible PTO.

• Benefits program.