Staff Research Engineer – Post-training & Evaluation

This is a fully remote position, open to applicants in United States.

📋 Description

• Establish the 'Reddit Benchmark' evaluation standard: Take ownership of the methodology — not merely the tools — for systematically assessing model quality across Safety, Reasoning, representation/retrieval, and Reddit-specific knowledge. Define measurable criteria for what 'Reddit-native' entails and set the benchmark for the organization.

• Ensure evaluation reliability and statistical precision: Develop the scientific basis for dependable evaluations — analyze variance, multi-sample scoring, inter-rater/inter-sample concordance, sampling and temperature influences, and the calibration of automated evaluators. You will be responsible for determining whether a benchmark delta is genuine or merely noise. Promote the use of evaluations as a release gate — conducted offline on static datasets, and pre-merge in CI/CD — to identify regressions prior to endpoint deployment.

• Create the model-as-a-judge methodology: Manage judge selection, prompt creation, calibration, and reliability for automated evaluations utilizing cutting-edge external models, facilitating swift and reliable iteration cycles.

• Develop post-training strategies and recipes: Craft SFT recipes (data mixtures, curriculum, and ablation strategies) that transform base models into useful, well-aligned endpoints; collaborate with engineering teams to scale these solutions.

• Assess base and CPT checkpoints, beyond just endpoints: Formulate methodology for selecting checkpoints across CPT experiments and LR studies, ensuring the right base model is chosen before committing to post-training computing resources.

• Lead synthetic data generation initiatives: Define and curate high-quality instruction and evaluation datasets to enhance generalization in areas where human data is limited.

• Collaborate with Safety Engineering: Translate overarching safety policies into specific classification metrics, probe sets, and CI/CD unit tests — including precision/recall at thresholds, label-noise management, and false-positive classifications for abuse detection (HHV).

• Investigate post-training instability: Analyze loss curves and evaluation logs to pinpoint alignment issues and capability degradation, and suggest solutions.

• Direct research initiatives: Establish the technical pathway for evaluation and post-training within the team, mentor engineers and scientists, and represent the team's work both internally and externally when suitable.

⛳️ Requirements

• A minimum of 6 years of professional experience in ML (or a PhD plus 4 years) specifically focused on LLM post-training and evaluation.

• A PhD or MS in Computer Science, Machine Learning, Natural Language Processing, Information Retrieval, or a closely related quantitative discipline — or equivalent experience in industry research.

• Extensive knowledge of evaluation reliability: including judge/sample variance, multi-sample scoring, calibration, statistical significance, and the limitations of automated evaluation methods.

• Proven experience in building tailored, domain-specific evaluation frameworks (e.g., lm-eval-harness, Inspect AI, LightEval) — with a solid understanding of the strengths and weaknesses of benchmarks like MMLU and GSM8K, and when they are inapplicable, treating evaluation sets as version-controlled, frozen, regression-tracked code.

• Experience in evaluating both generation and representation/classification: utilizing model-as-a-judge for generative quality and precision/recall, PR-AUC, retrieval/MTEB-style metrics, gold-label denoising, and label-noise management.

• In-depth knowledge of Continuous Pre-training (CPT), Instruction Tuning (SFT), and the influence of data quality on model behavior.

• Proficiency in Python; strong engineering skills in data pipeline and evaluation harness (e.g., Hugging Face Transformers, vLLM, lm-eval-harness). Familiarity with PyTorch and distributed training (FSDP2, DeepSpeed ZeRO-3) sufficient for directing and troubleshooting post-training processes.

🏝️ Benefits

• Comprehensive Healthcare Benefits and Income Replacement Programs

• 401k with Employer Match

• Global Benefit programs that accommodate your lifestyle, encompassing workspace, professional development, and caregiving support

• Family Planning Support

• Gender-Affirming Care

• Mental Health & Coaching Benefits

• Flexible Vacation & Paid Volunteer Time Off

• Generous Paid Parental Leave