Open Source Software Engineer

This is a fully remote position, open to applicants in New York.

📋 Description

• Evaluate software-engineering benchmark tasks for accuracy, reproducibility, and grading integrity.

• Conduct audits of repository-level assignments, reference patches, test harnesses, containerized environments, and evaluation logic.

• Analyze task requirements, state of the repository, dependencies, configuration, expected outcomes, and reproducibility.

• Examine reference patches for accuracy and completeness, while identifying unintended behavioral changes or incomplete fixes.

• Review test runners and automated grading logic for coverage, fragile assertions, and scoring consistency.

• Confirm the installation of dependencies, builds, configuration, runtime requirements, Docker isolation, and containerized execution.

• Detect answer leakage, unintended shortcuts, reward-hacking opportunities, loopholes, and exploitable behaviors.

• Investigate failing or inconsistent benchmark tasks using stack traces, logs, test failures, and repository behavior.

• Utilize open-source contribution and maintainer experience to assess repository conventions, patches, and workflows.

• Analyze Python software and tasks that involve Java, Go, TypeScript, or C++.

• Evaluate benchmark tasks against structured technical criteria and provide detailed written explanations with references to code, tests, patches, or execution.

• Differentiate between significant benchmark defects and minor implementation variations, recommending enhancements.

⛳️ Requirements

• A minimum of 3 years of professional software engineering experience.

• Proven experience in open-source contribution or maintainer roles, such as merged pull requests, committer responsibilities, or maintainer positions.

• Strong capability to review software changes at the repository level.

• Experience in auditing reference patches, test runners, and automated test suites.

• Proficient in evaluating Docker-based isolation and reproducible development environments.

• Solid understanding of software testing, debugging, and code review methodologies.

• Ability to identify issues related to answer leakage, reward hacking, or other benchmark-integrity concerns.

• High proficiency in Python programming.

• Professional fluency in at least one additional programming language, such as Java, Go, TypeScript, or C++.

• Familiarity with SWE-Bench Verified or similar repository-level software engineering benchmarks is a plus.

• A history of maintaining or contributing to established Python open-source projects is highly regarded.

• Prior experience in code review, software evaluation, or task grading is advantageous.

• Excellent written communication skills and the ability to provide detailed technical feedback.

• Work must be conducted without using confidential or proprietary information from any employer, client, institution, or third party.

• H1-B and STEM OPT support is not available for this engagement.

🏝️ Benefits

• Engagement as a part-time independent contractor.

• Fully remote work within the United States.

• Flexible scheduling based on project requirements.

• Projects may be extended, shortened, or concluded based on project needs and performance.