Senior Software Engineer – Open Source, SWE-Bench Evaluation

This is a fully remote position, open to applicants in Argentina.

📋 Description

• Analyze software engineering tasks that originate from actual GitHub issues and pull requests

• Evaluate the clarity and completeness of problem statements and success criteria

• Review unit tests for accuracy, coverage, and robustness

• Detect flaky tests, missing dependencies, version conflicts, and environmental issues

• Ascertain if tasks can be consistently reproduced across different environments

• Assess the real-world difficulty and complexity associated with each task

• Offer clear recommendations on whether tasks should be accepted, enhanced, or disregarded

• Review genuine bug fixes and feature implementations, repository configurations, dependency management, multi-file and cross-module changes, along with technical quality

⛳️ Requirements

• Over 3 years of professional experience in software engineering

• Significant experience with large, multi-file codebases

• Proficient in reviewing pull requests, debugging issues, and maintaining production-level code

• Strong grasp of unit testing and test coverage principles

• Capability to assess whether tests accurately validate a solution without unnecessarily limiting implementation options

• Familiarity with dependency management, environment setup, and reproducibility

• In-depth understanding of Git and GitHub-based development workflows

• Ability to analyze complex technical challenges and provide articulate written feedback

• Contributions to or maintenance of open-source projects (preferred)

• Experience with SWE-Bench, SWE-Bench Verified, or similar coding benchmarks (preferred)

• Familiarity with major Python open-source projects such as Django, Flask, scikit-learn, SymPy, matplotlib, requests, or pytest (preferred)

• Experience with Docker, CI/CD, pip, conda, or dependency pinning (preferred)

• Knowledge of test fixtures, test isolation, or property-based testing (preferred)

• Experience in designing technical assessments or evaluating coding challenges (preferred)

• Background in AI/ML evaluation, data curation, RLHF, or benchmark development (preferred)

🏝️ Benefits

• Flexible remote work options

• Part-time, project-based consulting engagement