Researcher – Benchmark Reviews

This is a fully remote position, open to applicants in United States.

📋 Description

• Evaluate benchmarks: Conduct an assessment of a new benchmark at least biweekly. You will analyze their methodology and determine what constitutes 'good performance' in relation to AI capabilities.

• Generate and update reports: Create public-facing reports on new benchmarks and regularly revise them as new versions are released and models advance.

• Analyze data: Investigate individual tasks within benchmarks thoroughly, utilizing coding agents for assistance while maintaining your own critical oversight.

⛳️ Requirements

• You possess exceptional written communication skills. Your initial drafts are typically clear and of publication quality, requiring minimal revisions.

• You critically assess research methodologies. You have strong viewpoints on the ideal characteristics of benchmarks and evaluations, and can identify when something appears incorrect.

• You are already acquainted with existing AI benchmarks, including their methodologies, advantages, and limitations. You have concepts for new benchmarks that currently do not exist but should.

• You are adept at working with complex data. You will often break down benchmarks into their fundamental components, which can be challenging without prior experience.

• Experience in writing about AI/ML for public or research audiences is a plus.

• Practical experience in conducting or developing AI evaluations is also a plus.

🏝️ Benefits

• Annual salary ranging from $100,000 to $200,000 USD, contingent on location, seniority, and experience.

• Salaries are not limited to USD; contracts and payments are typically made in local currencies.

• Fully remote work environment, with flexible working hours.

• Competitive global benefits package, including a comprehensive health insurance program—featuring supplemental benefits tailored to local regulations—and life insurance and a pension plan, where applicable.

• Generous paid time off (PTO), with no specified annual limit, including 30 days of PTO per year guaranteed, unlimited personal and sick leave, and 4 months of paid parental leave for permanent employees with a minimum of 12 months of service.

• A flexible and generous expense policy that allows for spending on equipment and a wide array of productivity tools or learning and development opportunities, including unlimited expenditure on AI tools, subject to regulations and managerial approval.

• Funded work trips, including 3 staff retreats annually and relevant conferences.

• Access to our well-equipped offices in Berkeley, California, which include complimentary meals, snacks, gym facilities, and more.