This is a fully remote position, open to applicants in United States.

📋 Description

• Develop and sustain cybersecurity benchmarks for AI within the evaluation infrastructure, primarily utilizing the Inspect library.

• Propose and prototype innovative ideas for new benchmarks and experiments.

• Ensure the benchmark suite operates smoothly to allow for prompt evaluations of new model releases.

• Sandbox and fortify evaluation environments against advanced AI models.

• Observe the interactions between AI agents and coding harnesses within various environments.

• Apply threat modeling to guide benchmarking strategies and measurement choices.

• Contribute strategic insights regarding AI cybersecurity evaluation tactics.

• Collaborate with AI providers and set up benchmarks to function on the infrastructure.

• Work alongside researchers, analysts, and engineers to guarantee the accuracy, insightfulness, and integration of evaluation data and outputs into research products and publications.

• Report directly to the CTO.

⛳️ Requirements

• Extensive software engineering experience, including several years of building and maintaining intricate systems while consistently contributing robust, maintainable code.

• Experience in sandboxing software against highly skilled adversaries.

• Red team experience in ethically hacking software.

• Familiarity with agentic AI usage and coding harnesses.

• Capability to generate creative ideas for new benchmarks and experiments.

• Driven by Epoch AI’s mission to deliver rigorous, independent insights into critical trends in AI.

• Professional proficiency in English.

• Applications must be submitted in English.

• Experience in benchmarking AIs and conducting evaluations (preferred).

• Experience in establishing CTF infrastructure (preferred).

• Experience utilizing modern AI for red teaming or ethical hacking (preferred).

• Knowledge of how AI coding harnesses and tool calls function (preferred).

• Familiarity with inspect_ai (preferred).