Senior Software Engineer – Open Source, SWE-Bench Evaluation

This is a fully remote position, open to applicants in Argentina.

📋 Description

• Analyze machine learning challenges related to experiment design, model selection, datasets, metrics, preprocessing, distribution shifts, contamination, label noise, feature leakage, hyperparameter tuning, and train/validation/test methodologies, as well as reproducibility and statistical significance.

• Assess whether the challenges are technically valid, reproducible, appropriately challenging, and require robust machine learning reasoning.

• Evaluate if datasets contain significant and learnable signals.

• Identify unintended shortcuts or artifacts present in synthetic datasets.

• Ascertain whether tasks necessitate a genuine diagnosis of underlying machine learning issues instead of relying solely on brute-force model selection or extensive hyperparameter searches.

• Review the evaluation metrics and improvement thresholds.

• Detect instances of metric manipulation, data leakage, and evaluation flaws.

• Confirm reproducibility throughout the entire data-to-model-to-evaluation pipeline.

• Evaluate whether the difficulty of challenges is appropriately calibrated.

• Offer recommendations for enhancing, recalibrating, or omitting problematic tasks.

• Analyze machine learning experiments, datasets, metrics, and pipelines related to applied machine learning model training and evaluation challenges.

⛳️ Requirements

• Over 3 years of practical experience in applied machine learning.

• Extensive experience in ML experiment design, model selection, hyperparameter tuning, model evaluation, data preprocessing, and validation.

• Deep understanding of train, validation, and test data splits.

• Proficiency in identifying data leakage, label noise, distribution shifts, spurious correlations, feature leakage, and data contamination.

• Experience in evaluating whether performance enhancements are statistically significant rather than mere random variations.

• Strong grasp of machine learning evaluation metrics and the appropriate contexts for their application.

• Proven experience in debugging machine learning workloads in both CPU and GPU environments.

• Ability to analyze technical issues and provide clear, written feedback.

• Experience in participating or contributing to ML competitions such as Kaggle or DrivenData is a plus.

• Experience in designing benchmark datasets or ML challenges is advantageous.

• Background in data-centric AI or dataset quality is a plus.

• Familiarity with synthetic data generation and validation is beneficial.

• Knowledge of statistical testing, confidence intervals, and effect sizes is a plus.

• Experience with ML evaluation pipelines, RLHF, or AI model evaluation is desirable.

• Experience in developing ML curricula or technical assessments is a bonus.

• Understanding concepts like shortcut learning, spurious correlations, Goodhart’s Law, Simpson’s paradox, and metric manipulation is advantageous.

• Applicants are required to choose a programming language or library for the interview and provide a location.

🏝️ Benefits

• Opportunity for remote work.

• Part-time, project-based consulting engagement.