Lead Engineer, AI Platform

This is a fully remote position, open to applicants in Europe.

📋 Description

• Develop and oversee evaluation infrastructure, encompassing CI/CD pipelines, scoring systems, and datasets for Circle’s AI agents.

• Identify and troubleshoot failures throughout the agent pipeline, which includes planning, execution, tool selection, and trajectories.

• Transform diagnostic insights into prototypes or priorities for the AI Core engineering team.

• Create annotation workflows and generate AI-produced and simulated conversations for evaluation datasets.

• Conduct structured experiments on prompts, models, and the agent harness.

• Assess new and open-source models in comparison to production baselines.

• Construct frameworks for model-shift decisions and enhance cost-effectiveness and latency through model swaps, caching, and routing.

• Lead and guide the AI Quality engineering team while remaining actively involved in coding.

• Define technical direction and oversee daily priorities.

• Collaborate with AI Core engineering to confirm quality enhancements.

⛳️ Requirements

• Over 7 years of experience in developing and deploying production software, preferably including LLM-powered agents that perform real actions within a product.

• Familiarity with complex, tool-utilizing systems, which involve multiple tools, planning or orchestration, and sub-agents.

• Proficient in Ruby on Rails / Python or willing to learn them rapidly.

• Strong proficiency in Python is highly advantageous, particularly for data and evaluation tasks.

• Experience in creating evaluation or observability infrastructure for ML/AI systems.

• Knowledge of evaluation frameworks such as Braintrust, LangSmith, or similar tools.

• Experience in designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.

• Quick learner, willing to experiment, and able to thrive in a fast-paced, dynamic environment.

• Comfortable working amidst ambiguity in a high-speed setting.

• Strong alignment with company values.

• Fluent in English, both spoken and written, at CEFR Level C2 / ILR Level 5.

🏝️ Benefits

• Equity in the company with ongoing refresh grants.

• 35 days of paid time off annually.

• Remote-only work arrangements with flexibility and autonomy.

• Company retreats held twice a year in amazing locations around the globe.

• Comprehensive benefits package that supports health, wellbeing, and professional development.