Lead Engineer, AI Platform

This is a fully remote position, open to applicants in Europe.

📋 Description

• Develop and oversee evaluation infrastructure for Circle’s AI agents.

• Create CI/CD pipelines, scoring mechanisms, and datasets to assess agent performance.

• Identify failures within agent pipelines, encompassing planning, execution, tool selection, and tool trajectories.

• Transform diagnostic insights into prototypes or priorities for AI Core engineering.

• Construct annotation workflows and facilitate AI-generated or simulated conversations.

• Conduct structured experiments across prompts, models, and the agent harness.

• Assess new and open-source models against existing production benchmarks.

• Enhance cost efficiency and latency through model swaps, caching strategies, and routing based on plan complexity.

• Lead and manage the AI Quality engineering team while maintaining hands-on involvement in coding.

• Establish technical direction and oversee daily priorities.

• Collaborate with AI Core engineering to verify quality enhancements.

⛳️ Requirements

• Over 7 years of experience in building and deploying production software.

• Background in LLM-powered agents that perform real actions within a product.

• Experience with intricate tool-using systems, including multiple tools, planning or orchestration, and sub-agents.

• Proficient in Ruby on Rails and Python, or capable of quickly acquiring these skills.

• Experience in developing evaluation or observability infrastructure for ML/AI systems.

• Familiarity with evaluation pipelines, scorers, dashboards, or CI/CD processes for evaluations.

• Experience with evaluation frameworks like Braintrust, LangSmith, or similar tools.

• Skilled in designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.

• Ability to rapidly learn and experiment in a dynamic environment.

• Comfort in navigating a fast-paced environment with uncertainty.

• Strong alignment with organizational values.

• Proficient in English at CEFR Level C2 / ILR Level 5.

🏝️ Benefits

• Equity in the company with ongoing refresh grants.

• 35 days of paid time off annually.

• Remote-only work environment.

• Exceptional flexibility and autonomy.

• Comprehensive benefits package supporting health, wellbeing, and professional development.

• Biannual company retreats/offsites.

• U.S.-benchmarked compensation offered globally.