Senior Software Engineer – Open Source, SWE-Bench Evaluation

This is a fully remote position, open to applicants in Argentina.

📋 Description

• Analyze software engineering assignments derived from actual GitHub issues and pull requests.

• Evaluate genuine bug fixes and feature implementations.

• Review open-source repositories, pull requests, unit tests, and test coverage.

• Assess repository configurations and dependency management.

• Analyze reproducibility and environment settings.

• Evaluate task complexity, difficulty, and code modifications that span multiple files or modules.

• Determine if tasks are well-defined, technically feasible, sufficiently tested, and reflective of professional software engineering challenges.

• Identify flaky tests, absent dependencies, unclear requirements, version conflicts, and environment-specific behaviors.

• Offer clear recommendations on whether tasks should be accepted, enhanced, or disregarded.

• Provide technical insights and quality evaluations.

⛳️ Requirements

• Minimum of 3 years of professional software engineering experience.

• Strong familiarity with extensive, multi-file codebases.

• Proven experience in reviewing pull requests, troubleshooting issues, and maintaining production code.

• Solid understanding of unit testing and test coverage principles.

• Capability to assess if tests adequately validate a solution without unduly limiting implementation strategies.

• Experience in dependency management, environment preparation, and reproducibility.

• Comprehensive understanding of Git and GitHub development workflows.

• Ability to dissect intricate technical issues and provide clear written feedback.

• Contributions to or management of open-source projects (preferable).

• Familiarity with SWE-Bench, SWE-Bench Verified, or comparable coding benchmarks (preferable).

• Experience with significant Python open-source projects such as Django, Flask, scikit-learn, SymPy, matplotlib, requests, or pytest (preferable).

• Experience with Docker, CI/CD, pip, conda, or dependency pinning (preferable).

• Knowledge of test fixtures, test isolation, or property-based testing (preferable).

• Experience in crafting technical assessments or reviewing coding challenges (preferable).

• Background in AI/ML evaluation, data curation, RLHF, or benchmark development (preferable).

🏝️ Benefits

• Compensation of $65 per hour.

• Opportunity for remote work.

• Part-time, project-based consulting role.