This is a fully remote position, open to applicants in United States.

📋 Description

• Choose tasks from a curated selection of validated Python repositories linked to specific git commits.

• Investigate repositories and pertinent PR or issue histories to gain insight into the target subsystem.

• Reformulate questions to test frontier coding agents while keeping them clear and answerable.

• Modify rubrics and distractors to reward accurate reasoning and eliminate plausible-but-incorrect answers.

• Execute the complete Studio validation loop: Check, Golden, Run, Analyze.

• Record efforts in task READMEs and submit tasks for expert evaluation.

• Respond to reviewer comments on returned tasks.

• Create datasets that train AI agents to reason about authentic Python codebases.

⛳️ Requirements

• Over 3 years of professional software engineering experience, particularly in production Python.

• Capability to navigate extensive, unfamiliar Python codebases and articulate runtime behavior.

• Proficiency in Python internals, encompassing import machinery, descriptors and metaclasses, async and event loops, packaging, and C-extension boundaries.

• Experience with web frameworks, developer tools, type checkers, or scientific/numeric libraries is highly pertinent.

• Strong command of written English.

• Comfortable using git at the commit level, along with containerized environments and command-line tools.

• Must undergo identity verification, background check, sign Terms of Work and CIIAA, complete a tax form, and a brief calibration quiz prior to task access.

• Qualified candidates may be requested to complete a short technical assessment or provide further information.