Senior Software Engineer, AI Evaluation Infrastructure

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop and deploy comprehensive features across the data model, API, background processes, and user interface.

• Enhance agent and model assessments within isolated environments.

• Evaluate scores and transform thousands of runs into reliable outcomes.

• Elevate benchmark task quality assurance through verifiers, rubric-based grading, and LLM-as-a-judge evaluations.

• Incorporate new models, agents, and benchmarks via flexible design strategies.

• Boost system performance and dependability through optimized queries, secure migrations, and effective concurrency management.

• Assess colleagues’ pull requests in accordance with engineering standards.

• Deliver well-tested, maintainable updates while independently managing specific system components.

⛳️ Requirements

• Generally requires 5+ years of experience in building and managing production backend systems.

• Responsible for at least one service from inception to on-call support.

• Proficient in Python.

• Familiarity with Django or a comparable framework, PostgreSQL, and a task queue like Celery.

• Practical experience in LLM evaluation, including the creation of benchmark tasks, scoring agent trials, and aggregating/comparing results.

• Ability to assess task quality, addressing issues such as ambiguous instructions, ineffective or exploitable verifiers, leaked answers, and invalid checks.

• Capability to make informed trade-offs and select the most efficient design.

• Responsible for migrations, testing, and handling edge cases.

• Quick to adopt team conventions for naming, structure, testing, and error management.

• Comfortable with TypeScript and engaging in React frontend development.

• Demonstrated ability in clear technical writing for pull request descriptions and design documentation.

• Preferred skills: Docker, Kubernetes, Modal, Daytona, LLM-as-a-judge, rubric-based grading, Claude Code, Codex, OpenHands, MCP, distributed job scheduling, concurrency limits, database locking at scale, and contributions to open-source evaluation or infrastructure.

🏝️ Benefits

• Work at the cutting edge of AI, assisting leading AI laboratories in enhancing advanced models by creating expert datasets, reinforcement learning environments, and benchmarks.

• Engage in pioneering AI research and have the opportunity to present your work at conferences such as ICLR, ICML, and NeurIPS.

• Apply frontier AI innovations to address enterprise challenges.

• Collaborate with and learn from peers with extensive AI experience from top companies like Google, Meta, and Amazon.

• Experience a fast-paced startup environment characterized by speed, ownership, and significant impact.

• Join a workplace that values equal opportunity, diversity, inclusion, and authenticity.