Software Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Create and deploy comprehensive features encompassing the data model, API, background services, and user interface.

• Develop and enhance workflows for evaluating agents and language models through sandboxed trials, scoring, and the aggregation of extensive results.

• Elevate the quality checks for benchmark tasks by utilizing verifiers, rubric-based grading, and LLM-as-a-judge evaluations.

• Incorporate new models, agents, and benchmarks via adaptable systems.

• Ensure performance and dependability through optimized queries, secure migrations, and scalable concurrency.

• Evaluate pull requests according to engineering standards and uphold consistency within the codebase.

• Implement multiple changes within the first month while confidently taking ownership of a specific system area.

⛳️ Requirements

• Over 5 years of experience in building and managing production backend systems, including the responsibility for at least one service from design to on-call duties.

• A Bachelor’s degree or higher in Software Engineering, Computer Science, or a related discipline.

• Proficient in Python with experience in Django or a comparable framework, PostgreSQL, and a task queue like Celery.

• Practical experience in LLM evaluation beyond merely calling a model API.

• Knowledge of benchmark-task creation, agent-trial scoring, and the aggregation and comparison of results.

• Acumen regarding task quality, including handling ambiguous instructions, ineffective or manipulable verifiers, leaked answers, and invalid checks.

• Capability to make practical design compromises and articulate what was not implemented.

• Responsibility for end-to-end delivery, encompassing migrations, testing, and edge case handling.

• Quick adaptation to team conventions for naming, structure, testing, and error management.

• Proficiency in TypeScript and experience with React for frontend development.

• Clear technical writing skills in pull request descriptions and design documentation.

• Familiarity with Harbor, Terminal-Bench, SWE-bench, Inspect, or similar testing harnesses is an advantage.

• Experience with Docker, Kubernetes, Modal, or Daytona is a plus.

• Background in LLM-as-a-judge, rubric-based grading, and minimizing judge bias and noise is an asset.

• Knowledge of Claude Code, Codex, OpenHands, and MCP is beneficial.

• Experience with distributed job scheduling, concurrency limitations, or database locking at scale is an advantage.

• Contributions to open-source evaluation or infrastructure are a plus.

🏝️ Benefits

• Equity.

• Opportunity to work at the cutting edge of AI alongside leading AI labs.

• Chances to contribute to pioneering AI research and present work at ICLR, ICML, and NeurIPS.

• Opportunity to leverage frontier AI innovations to tackle enterprise challenges.

• Collaboration and learning alongside exceptional colleagues experienced in AI.

• A startup-speed environment offering ownership and significant impact.