Python Engineer, AI Coding Agent Evaluator

This is a fully remote position, open to applicants in United States.

📋 Description

• Conduct comprehensive evaluations of AI-generated coding interactions from start to finish.

• Analyze the coherence of responses and the usefulness of preambles and reasoning.

• Assess whether outputs exhibit strong engineering judgment.

• Determine if interactions are suitable for seasoned developers.

• Evaluate the usefulness, overall correctness, and alignment with sound engineering principles.

• Assess explanations and reasoning in addition to the code itself.

• Differentiate between varying levels of response quality.

• Provide clear and decisive feedback on what was effective, what didn't work, and what seemed misleading.

• Assist in defining the characteristics of excellent interactions for tools like Cursor.

• Make informed yet subjective judgments regarding the behavior of AI coding agents.

⛳️ Requirements

• Highly skilled software engineer at a Senior+ level.

• Experience equivalent to a Staff / Principal-level engineer.

• Strong proficiency in TypeScript / JavaScript or Python.

• Practical experience with OpenAI Codex, Claude Code, and Cursor.

• Significant familiarity with contemporary AI-assisted development workflows.

• Capable of evaluating code without the necessity of executing or thoroughly reviewing every line.

• Comfortable providing direct, opinionated feedback.

• High standards for engineering quality.

• Complete a take-home evaluation exercise.

• Participate in one behavioral interview.

• Undergo a straightforward background check for the project.

• Required to choose preferred programming languages from the available list.

🏝️ Benefits

• Hourly rate between $100 and $200.

• Approximately 10 to 20 hours of work per week.

• Potential for extension beyond early May.