Staff Software Engineer, AI Evals

This is a fully remote position, open to applicants in United States.

📋 Description

• Conduct comprehensive evaluations of AI-generated coding interactions from start to finish.

• Analyze the sensibility of responses and the utility of preambles and reasoning.

• Assess whether the outputs demonstrate sound engineering judgment, reflecting the thought processes of an experienced engineer.

• Evaluate the quality of explanations and reasoning beyond just the code itself.

• Differentiate between various levels of response quality.

• Deliver clear, opinionated feedback on effective elements, shortcomings, and any misleading aspects.

• Assist in defining exemplary interactions for tools such as Cursor.

• Review interactions with contemporary coding agents, including OpenAI Codex and Claude Code.

⛳️ Requirements

• Staff or Principal-level engineer (or equivalent experience).

• Strong proficiency in Go, Python, Rust, or TypeScript/JavaScript.

• Practical experience with OpenAI Codex, Claude Code, and Cursor.

• Deep understanding of modern AI-assisted development workflows.

• Capability to evaluate code without the necessity to execute or deeply scrutinize every line.

• Comfort in providing direct, opinionated feedback.

• High standards for what constitutes “good engineering.”

• Ability to make subjective yet rigorous assessments.

• Experience with Cursor or similar AI-first Integrated Development Environments (preferred).

• Prior exposure to prompt design or evaluation processes (preferred).

• Experience in mentoring senior engineers or establishing engineering standards (preferred).

• Native English speaker or fluent in English at the C2+ level.

• Willingness to undergo a simple background check.

• Completion of a take-home evaluation exercise is required.

🏝️ Benefits

• Compensation ranging from $100 to $200 per hour.

• Flexible workload of approximately 20 to 40 hours per week.

• Contract duration through October with the possibility of extensions.

• Immediate start available.