Data Annotation Specialist, Software Engineering

This is a fully remote position, open to applicants in Canada.

📋 Description

• Analyze the model's performance in handling coding requests, workflows, and inquiries related to the code base using available tools.

• Evaluate agent trajectories and the model's capabilities for generating and debugging code.

• Engage models to tackle intricate coding tasks and assess the precision of the produced responses.

• Annotate, edit, and enhance both machine-generated and human-created outputs in software engineering.

• Document quality and performance trends associated with model/agent behavior and project assignments.

⛳️ Requirements

• Proven expertise in software engineering, backed by both academic study and 3-5 years of relevant professional experience.

• Strong knowledge and comprehension of Python, as well as familiarity with the following programming languages: Java, JavaScript, Go, and SQL (any dialect).

• Previous experience in designing, building, and deploying APIs.

• Prior work with code agents (such as OpenCode, Claude Code, Codex, Cursor) or assessing agent trajectories is a significant advantage.

🏝️ Benefits

• Performance incentives included!