Staff AI Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Define the technical direction for agent systems across various teams, focusing on orchestration, tool utilization, context and memory management, human oversight, and failure recovery.

• Develop, construct, and maintain shared foundations for evaluations, observability, tracing, execution controls, and model access with routing and failover capabilities.

• Implement evaluation methodologies for task completion, accuracy, false positives, security, and cost per successful task.

• Collaborate with domain experts to create representative test cases and convert production failures into regression tests.

• Enhance resilience against tool failures, timeouts, rate limits, and partial executions by enforcing limits on retries, runtime, and expenditure.

• Establish scoped permissions, ensure customer data isolation, create audit trails, and facilitate safe tool/code execution in collaboration with security and platform teams.

• Set engineering standards for continuous integration evaluations, prompt/model versioning, regression testing, cost and latency budgets, and staged rollouts.

• Drive model selection and cost optimization by benchmarking models against actual workloads.

• Collaborate with teams to ensure production readiness, conduct incident investigations, and enhance agent reliability.

• Assist engineering teams in utilizing AI coding agents throughout planning, implementation, review, and testing phases.

• Mentor engineers and technical leads through design reviews, collaborative work, internal presentations, and written documentation.

• Develop a technical roadmap in partnership with product, research, and security leaders.

⛳️ Requirements

• Extensive software engineering experience, typically 8–10+ years, with a proven history of technical leadership across teams.

• Practical experience in building and operating LLM-based agents in a production environment.

• Demonstrated success in enhancing agent reliability through evaluations, investigating non-deterministic failures, and achieving measurable improvements in success rates.

• In-depth knowledge of agent architectures, including tool utilization, orchestration, planning, context, and memory management.

• Experience in executing LLM workloads via managed cloud model platforms or direct provider APIs, including handling rate limits, failures, and cost tracking.

• Strong foundations in backend and distributed systems, encompassing APIs, asynchronous workflows, concurrency, and fault tolerance.

• Familiarity with production observability, incident investigation, and secure rollout of system modifications.

• Awareness of security risks associated with agent systems, including prompt injection, unsafe tool execution, and sensitive data exposure.

• Proven impact beyond individual coding contributions, such as development of shared platforms, enhancement of standards, or mentoring of engineers.

• Experience with agent orchestration, tool invocation, and structured outputs.

• Knowledge of Model Context Protocol (MCP), APIs, and permission-controlled tool execution.

• Experience with automated evaluation harnesses, domain-specific test datasets, and regression testing within CI.

• Proficiency in distributed tracing, structured logging, and metrics for multi-step agent workflows.

• Experience with managed cloud model platforms, direct provider APIs, and multi-provider gateways.

🏝️ Benefits

• Competitive salary and a compelling equity package.

• 401(k) plan (US) or pension scheme (EU).

• Medical, dental, vision, and life insurance (US) or statutory healthcare (EU).

• Wellness stipend.

• Work-from-home equipment and Wi-Fi allowance.

• Learning and development funding.

• Flexible and generous paid time off.

• Paid parental leave.

• Continuous mentorship opportunities.