Software Engineering Evaluator, Contract

This is a fully remote position, open to applicants in United States, +3 more locations.

📋 Description

• Develop grading criteria tailored for specific tasks such as feature implementations, bug fixes, refactoring, testing, and code reviews.

• Analyze AI-generated code for accuracy, including situations where failures still pass the code's own tests.

• Evaluate coding deliverables on their handling of edge cases, security, performance, and readability.

• Compose clear, evidence-based explanations for each evaluation score assigned.

• Maintain consistent judgment throughout software engineering evaluation tasks.

• Examine code that appears correct and operates without errors while identifying subtle flaws.

• Navigate ambiguity and clearly differentiate between established facts and inferences during evaluations.

• Refine evaluation processes based on feedback from senior reviewers.

⛳️ Requirements

• 2–3 years of professional experience in software engineering, specifically in shipping production code.

• Proficient in at least one of the following languages: Python, JavaScript/TypeScript, Java, Go, C++, C#, or a similar programming language.

• Strong foundation in data structures, algorithms, testing, debugging, and version control.

• Prior experience in reviewing code from other engineers.

• Excellent proficiency in written English.

• Capability to follow intricate instructions, reason through uncertainty, and identify subtle errors in code that appears correct and executes successfully.

• Ability to distinctly separate known information from inferences.

• Competence to work autonomously, rapidly learn new tools, and accept constructive feedback.

• Residing and working from the US, Canada, or UK.

• Availability for a minimum of 10 hours per week for the entire 12-week duration.

🏝️ Benefits

• Weekly payments through Stripe or Wise.

• Flexible remote work schedule within the 10+ hour weekly commitment.

• Initial contract for 12 weeks, with the potential for project extensions based on needs and performance.

• Opportunity to assess AI systems for leading organizations in AI research.

• Reasonable accommodations available upon request.