QA/Test Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Create detailed test scenarios for benchmark tasks, encompassing edge cases, failure conditions, and unexpected inputs.

• Assess task descriptions, requirements, expected outcomes, and reference solutions for clarity, inconsistencies, missing requirements, and quality concerns.

• Utilize Python and development tools to identify and resolve issues within task environments, validation logic, and automated checks.

• Develop repeatable testing procedures, quality checklists, and review frameworks.

• Analyze grading logic and AI-agent execution results for vulnerabilities, shortcuts, inconsistent scoring, and risks to benchmark reliability.

• Work collaboratively with researchers and task authors to address issues, suggest improvements, and verify fixes.

• Set and uphold quality standards across technical evaluation datasets.

⛳️ Requirements

• MSc or PhD in a STEM field, or equivalent practical experience in a research-driven or engineering-centric environment.

• At least 1 year of experience in QA, test engineering, software engineering, research engineering, or a similar quality-focused position.

• Capability to design effective test cases, develop QA processes, and analyze complex technical systems from start to finish.

• Proficient in Python and Git, with the ability to navigate unfamiliar codebases, environments, and technical workflows.

• Strong debugging and analytical abilities.

• Exceptional attention to detail and structured practices in technical documentation.

• Skillful at identifying subtle failures, inconsistencies, edge cases, and unintended behaviors.

• Experience in AI evaluation, AI training, model testing, benchmark development, or reviewing AI-generated outputs is highly preferred.

• Comfortable working independently on ambiguous and open-ended technical challenges.

• Excellent written communication skills and the ability to provide actionable feedback to technical stakeholders.

• Ability to consistently commit to approximately 35 hours per week.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Opportunities for professional development and continued education.

• Flexible working hours and remote work options.

• Collaborative and innovative work environment.

• Comprehensive health and wellness benefits.