Senior Python Engineer – LLM Code Evaluation

This is a fully remote position, open to applicants in United States.

📋 Description

• Analyze AI-generated coding interactions from start to finish.

• Evaluate whether the responses are beneficial, accurate at a high level, and reflect sound engineering judgment.

• Review the quality of explanations, introductions, and reasoning.

• Differentiate between various levels of response quality.

• Offer clear and constructive feedback on what succeeded, what failed, and what seemed misleading.

• Assist in defining the characteristics of exceptional interactions for tools like Cursor.

• Assess whether AI coding agents communicate and guide users as proficient engineers would.

• Evaluate interactions with OpenAI Codex and Claude Code instead of producing code directly.

⛳️ Requirements

• Extensive experience as a software engineer at a Senior+ level.

• Experience at a Staff / Principal-level engineer or similar qualifications.

• Strong expertise in TypeScript / JavaScript or Python.

• Practical experience with OpenAI Codex, Claude Code, and Cursor.

• In-depth understanding of contemporary AI-assisted development workflows.

• Capable of evaluating code without needing to execute or scrutinize every line thoroughly.

• Comfortable giving direct and opinionated feedback.

• High expectations for quality engineering standards.

• Completion of a take-home evaluation exercise.

• Participation in one behavioral interview.

• Consent to a straightforward background check.

• Availability to start immediately and work through early May.

• Preferred programming languages include Python and TypeScript from the provided list.

• Location information is required as part of the application process.

🏝️ Benefits

• Compensation ranging from $100 to $200 per hour.

• Potential for contract extension.

• Flexible part-time role requiring approximately 10 to 20 hours per week.