Member of Technical Staff, Evals

This is a fully remote position, open to applicants in California.

📋 Description

• Create new benchmarks and evaluation methodologies.

• Conduct data ablations.

• Evaluate the quality of datasets.

• Examine whether measurements accurately represent model performance.

• Design and validate evaluations and benchmarks throughout model training and deployment.

• Analyze model behavior and pinpoint deficiencies in existing evaluations.

• Discover opportunities to enhance training data.

• Enhance the correctness, reproducibility, and reliability of evaluations.

• Review and improve public benchmarks, research publications, and open-source implementations.

• Establish metrics that accurately signify model quality.

• Develop tools that render evaluation results accessible and beneficial across Magic.

⛳️ Requirements

• Proven experience in developing model evaluations, benchmarks, or machine learning experiments.

• Strong programming capabilities and familiarity with machine learning systems.

• Demonstrated history of managing technical projects from start to finish.

• A critical approach towards results that cannot be reproduced, validated, or explained.

• Background in analyzing model performance and interpreting experimental findings.

• Ability to navigate uncertainty and assess whether a measurement truly reflects the behavior it purports to measure.

• Dedication to enhancing the standards of rigor in AI evaluations.

🏝️ Benefits

• Equity constitutes a significant portion of total compensation, alongside the salary.

• 401(k) plan with a 6% salary matching.

• Comprehensive health, dental, and vision insurance for you and your dependents.

• Unlimited paid time off.

• Visa sponsorship and relocation assistance for candidates relocating to San Francisco.

• A small, agile, highly collaborative team focused on cutting-edge AI systems.