Member of Technical Staff, Benchmarking

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop and uphold clinical benchmarks for reasoning, comprehensive patient histories, multimodal records, and tool utilization.

• Construct end-to-end evaluation pipelines, encompassing runs, scoring, dashboards, and reports that compare models and training runs on a large scale.

• Design rubrics, automated grading systems, and LLM-as-a-judge frameworks, calibrating them against clinician assessments.

• Conduct systematic failure analyses, categorize failure modes, assess their frequency, and incorporate the findings into dataset design and post-training processes.

• Collaborate with clinical specialists and partner AI teams to translate benchmark outcomes into research publications and products.

• Create benchmarks and evaluation systems tailored for real-world healthcare data, utilizing insights to generate data that bridges model performance gaps.

⛳️ Requirements

• Hands-on experience in evaluating or training LLMs (using PyTorch, Hugging Face, or similar frameworks).

• Proficient in Python and data engineering.

• Familiarity with large tabular and text datasets.

• Knowledge of SQL/BigQuery is advantageous.

• Capability to design evaluations that assess significant outcomes and avoid shortcuts.

• A B.Sc. or higher in Computer Science, Machine Learning, Natural Language Processing, biomedical informatics, or a related discipline.

• Approximately 4 hours of daily overlap with Pacific Time (around 9 AM to 1 PM PT).

• LLM/NLP publications are preferred.

• Experience with clinical data or medical benchmarks is a plus.

• Familiarity with reinforcement learning from verifiable rewards, reward modeling, or post-training data is advantageous.

🏝️ Benefits

• Equity.

• A well-funded, revenue-generating company.

• Remote work arrangement.