Senior Research Engineer, AI Evaluation

atSystem Inc.RemoteUS flagNew YorkFull-timeAI Research ScientistMid-levelSenior$180k – $240k/year

Posted 1 day ago

This is a fully remote position, open to applicants in New York.

📋 Description

• Oversee the definition, implementation, and expansion of the evaluation architecture and algorithmic benchmarking systems for the System’s LLM.

• Engage in research and development that enhances System’s fundamental platform and product offerings.

• Design, construct, and uphold automated, reproducible evaluation frameworks to assess LLM reasoning, retrieval accuracy, and structured outputs across CDS applications.

• Develop both general and domain-specific evaluation benchmarks, synthetic test-case generators, and high-quality datasets.

• Measure and address edge cases, hallucinations, model drift, and safety regressions.

• Work collaboratively with AI Research Scientists, ML Infrastructure Engineers, and Product teams to create standardized offline and online evaluation loops integrated into CI/CD processes.

• Report directly to the Head of AI & Engineering.


⛳️ Requirements

• Minimum of 3 years' experience in software engineering, applied machine learning, or AI research, with a solid statistical foundation and scientific rigor.

• Proven experience in designing evaluation harnesses or benchmarking suites for LLMs and intricate reasoning systems (e.g., LLM-as-a-judge, reference-based metrics, human-in-the-loop review).

• Advanced proficiency in Python, along with experience using ML frameworks (PyTorch, Hugging Face), frontier model APIs, RAG pipelines, and experiment tracking/observability tools.

• Excellent written communication abilities (including journal article writing, blog posts, position papers, etc.).

• Applicants must possess legal authorization to work in the United States of America and must maintain ongoing work authorization throughout their employment.


🏝️ Benefits

• Meaningful equity ownership.

• Comprehensive health and wellness benefits aimed at supporting employee well-being.

• Generous paid time off for rest and rejuvenation.

• A flexible workplace policy featuring a hub in Lower Manhattan, along with full support for remote work.

People also viewed

Flare7 hours ago

Senior Threat Researcher, AI and Autonomous Cybercrime

US flagNew York OnlyFull-timeAI Research Scientist
ApplyView job
Helix10 hours ago

Senior Product Manager – AI Research Agent

US flagUnited States OnlyFull-timeAI Research Scientist$158k – $205.3k/year
ApplyView job
Weekday (YC W21)1 day ago

PhD Drug Discovery Scientist, AI Expert

US flagUnited States OnlyFull-timeAI Research Scientist$70 – $100/hour
ApplyView job
NVIDIA1 day ago

Senior Deep Learning Scientist, AI Security

US flagUnited States OnlyFull-timeAI Research Scientist$187.5k – $442k/year
ApplyView job
Prolific1 day ago

Director of AI Research

US flagUnited States, +3 more locationsFull-timeAI Research Scientist$275k – $400k/year
ApplyView job
NVIDIA2 days ago

Senior AI Research Engineer, Agentic Systems Intelligence

US flagCalifornia, +3 more locationsFull-timeAI Research Scientist$187.5k – $364k/year
ApplyView job

Never miss a great job!

Get handpicked remote jobs straight to your inbox weekly.

Trusted by 7,400+ designers