Senior Software Engineer, AI Benchmarking

This is a fully remote position, open to applicants in California, +1 more state.

📋 Description

• Conduct pre-release and post-release evaluations of frontier and open-source AI models utilizing SecureBio's comprehensive biosecurity assessment tools.

• Develop adapters for models, as well as identify and resolve unexpected agent behaviors.

• Create, scale, sustain, and perpetually enhance SecureBio's portfolio of biosecurity evaluations.

• Refine our assessments to incorporate cutting-edge agent frameworks, jailbreaking techniques, and elicitation strategies.

• Design and enhance internal tools that empower our researchers to conduct capability assessments at scale.

• Assist in the development and upkeep of cloud infrastructure required for executing our evaluations and analyses.

• Contribute insights to our public dashboard highlighting trends in AI biology capabilities.

⛳️ Requirements

• A demonstrated history in software engineering, with 5+ years of experience in industry positions and/or contributions to open-source projects.

• Proficiency in Python programming.

• Expertise in Docker and AWS, particularly within the ECS environment.

• Experience in developing software tools that utilize or assess LLMs or AI agents.

• Self-motivated and adept at working within small, agile teams.

• A dedication to sound coding practices, including understanding when and how to effectively use (or avoid using) coding agents.

• Emphasis on code quality, readability, and ensuring human oversight of both human- and AI-generated code.

• Alignment with SecureBio's mission to avert catastrophic pandemics.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Flexible work hours and opportunities for remote work.

• Professional development and continuous learning opportunities.

• Engaging and collaborative work environment focused on innovation.