QA Test Engineer

This is a fully remote position, open to applicants in New York.

📋 Description

• Develop thorough test cases to ensure benchmark tasks operate as expected

• Create tests that encompass positive, negative, boundary, and edge-case scenarios

• Verify task specifications, anticipated outcomes, reference solutions, and grading logic

• Detect scenarios that could yield inaccurate or misleading evaluation results

• Confirm that tests accurately assess the intended technical capabilities

• Review intricate multi-step tasks and reference solutions prior to finalization

• Spot ambiguous instructions, inconsistent requirements, missing assumptions, and incomplete acceptance criteria

• Execute tasks independently to validate reproducibility and anticipated behavior

• Evaluate whether grading standards are transparent, equitable, and technically justified

• Offer constructive feedback to task authors and researchers

• Investigate failures within Python scripts, test harnesses, repositories, and task environments

• Diagnose unexpected behaviors in unfamiliar codebases

• Reproduce reported issues and pinpoint their root causes

• Correct or document problems related to the environment, dependencies, logic, and validation

• Utilize Git-based workflows to facilitate structured review and collaboration

• Create practical checklists and repeatable review processes to ensure benchmark quality

• Enhance consistency across task validation, testing, and approval workflows

• Clearly document findings to enable authors to efficiently resolve issues

• Monitor recurring defects and suggest preventive quality measures

• Collaborate closely with researchers, task authors, and other technical reviewers

• Analyze AI agent runs for unintended shortcuts, loopholes, and grading weaknesses

• Identify instances where models can gain credit without completing the intended reasoning or technical tasks

• Test the robustness of benchmark tasks against alternative approaches

• Reinforce evaluation criteria to ensure reliable and meaningful benchmark scores

• Differentiate valid solution diversity from unintended exploitation of tasks

⛳️ Requirements

• Minimum of 1 year experience in test engineering, quality assurance, software engineering, research engineering, or a similar technical role

• Proven experience in designing test cases and quality-review processes

• Strong debugging skills across complex technical systems

• Proficient in Python and Git

• Comfortable navigating unfamiliar codebases, repositories, and execution environments

• Exceptional attention to detail and strong documentation skills

• Ability to recognize ambiguity, edge cases, hidden assumptions, and quality deficiencies

• Capability to work independently on open-ended technical challenges

• Reliable availability for approximately 35 hours per week

🏝️ Benefits

• Competitive salary and performance-based incentives

• Opportunities for professional development and training

• Flexible work hours and remote work options

• Inclusive and collaborative work environment

• Health and wellness benefits