This is a fully remote position, open to applicants in United States.

📋 Description

• Develop the evaluation framework for agent behavior, incorporating scenario suites derived from actual incidents, continuous integration regression gating, and metrics that effectively identify confidently incorrect agents.

• Collaborate with the Chief Architect to define the autonomy Service Level Objective (SLO) and take responsibility for the evidence needed to shift decisions from recommendations to actionable steps.

• Manage the agent architecture, which includes orchestration, tool dispatch, task decomposition, escalation procedures, and the governance of read/write boundaries.

• Enhance the explanations provided by agent reasoning to ensure that recommendations can be effectively implemented.

• Transform operational runbooks into reasoning that can be executed by agents.

⛳️ Requirements

• Experience in building and deploying LLM-based agentic systems in a production environment, addressing issues such as malformed tool outputs, loops, stagnant states, and context depletion.

• Possess strong and well-defined opinions regarding evaluation methodologies.

• Proven experience with tool and function invocation at scale.

• Familiarity with LLM tracing and online evaluation processes.

• Solid systems engineering principles in Python programming.

• Experience in multi-agent orchestration, handoff procedures, and context management.

• Preferred: background in operations or Site Reliability Engineering (SRE).

• Preferred: design experience with human-in-the-loop systems.

• Preferred: ability to retrieve information over operational data.

• Preferred: experience in defining the operational boundaries of autonomous systems.