Researcher, Evaluations

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop and curate a comprehensive evaluation suite. Identify real-world tasks that serve as robust tests for practical AI capabilities, and continuously update these tasks as AI technology progresses. Create evaluation rubrics for assessing AI performance.

• Assess AI systems. Regularly review new and significant AI models and products using the task suite. Revise tasks and rubrics to align with the evolving landscape of AI capabilities.

• Share your research findings. Produce public-facing reports, blog posts, and data visualizations that highlight your insights. Ensure that the evaluations contribute to our broader research initiatives and keep our team well-informed.

• Perform data analysis. Examine evaluation outcomes and compare models across various tasks.

• Optimize the evaluation process. You may automate certain aspects of the workflow and develop components of the evaluation into independent benchmarks.

⛳️ Requirements

• Strong analytical skills. You conduct experiments with diligence, ensuring that your findings are thoroughly supported by evidence.

• Critical and grounded mindset. You develop your own well-reasoned perspective on what an AI system can achieve, effectively differentiating between practical capabilities and hype.

• Proficiency with AI agents and tools. You have hands-on experience working with AI agents in your projects and are comfortable delegating tasks as needed.

• Knowledge of AI benchmarks and evaluations. You stay updated on AI capabilities and have informed opinions on the effectiveness of various benchmarks.

• Experience in research and data analysis, including sufficient coding skills to analyze your results independently.

• Excellent written communication abilities. You can articulate nuanced observations with clarity and precision.

🏝️ Benefits

• Annual salary range of **$115,000 – $200,000 USD**, commensurate with location and experience.

• Fully remote work environment with flexible working hours.

• Competitive global benefits package, including a comprehensive health insurance plan—featuring supplemental benefits specific to local regulations—and life insurance, along with a pension plan where applicable.

• Generous paid time off (PTO), with no specific annual limit, including a guaranteed 30 days of PTO per year, unlimited personal and sick leave, and 4 months of paid parental leave for permanent employees with at least 12 months of service (prorated for those with less than 12 months).

• A flexible and generous expense policy for purchasing equipment and a wide range of productivity tools or learning/development opportunities, including unlimited funding for AI tools, subject to regulations and managerial approval.

• Paid travel for work-related trips, including 3 staff retreats annually and attendance at relevant conferences.

• Access to our well-equipped offices in Berkeley, California, which include provided meals, snacks, gym facilities, and more. All staff, regardless of their location, can access the office for a minimum of 20 days each year.