
Senior Research Engineer, AI Evaluation
Posted 1 day ago

Posted 1 day ago
This is a fully remote position, open to applicants in New York.
• Oversee the definition, implementation, and expansion of the evaluation architecture and algorithmic benchmarking systems for the System’s LLM.
• Engage in research and development that enhances System’s fundamental platform and product offerings.
• Design, construct, and uphold automated, reproducible evaluation frameworks to assess LLM reasoning, retrieval accuracy, and structured outputs across CDS applications.
• Develop both general and domain-specific evaluation benchmarks, synthetic test-case generators, and high-quality datasets.
• Measure and address edge cases, hallucinations, model drift, and safety regressions.
• Work collaboratively with AI Research Scientists, ML Infrastructure Engineers, and Product teams to create standardized offline and online evaluation loops integrated into CI/CD processes.
• Report directly to the Head of AI & Engineering.
• Minimum of 3 years' experience in software engineering, applied machine learning, or AI research, with a solid statistical foundation and scientific rigor.
• Proven experience in designing evaluation harnesses or benchmarking suites for LLMs and intricate reasoning systems (e.g., LLM-as-a-judge, reference-based metrics, human-in-the-loop review).
• Advanced proficiency in Python, along with experience using ML frameworks (PyTorch, Hugging Face), frontier model APIs, RAG pipelines, and experiment tracking/observability tools.
• Excellent written communication abilities (including journal article writing, blog posts, position papers, etc.).
• Applicants must possess legal authorization to work in the United States of America and must maintain ongoing work authorization throughout their employment.
• Meaningful equity ownership.
• Comprehensive health and wellness benefits aimed at supporting employee well-being.
• Generous paid time off for rest and rejuvenation.
• A flexible workplace policy featuring a hub in Lower Manhattan, along with full support for remote work.
Flare
Helix
Weekday (YC W21)
NVIDIA
Get handpicked remote jobs straight to your inbox weekly.