Senior Manager, AI Performance & Operations

This is a fully remote position, open to applicants in United States.

📋 Description

• Take responsibility for the overall health of significant voice and text AI workflows, ensuring that business, quality, safety, and operational targets are achieved.

• Keep an updated overview of all key workflows, including their status, maturity levels, failure modes, and subsequent actions.

• Regularly monitor weekly indicators such as containment, escalation, and complaints, while managing production dashboards to independently track performance.

• Lead the response to AI regressions; identify whether issues originate from the model, prompt, or orchestration, and facilitate post-mortems to resolution.

• Conduct structured cycles (hypothesis, testing, rollout/rollback) for ‘mini-launches’ such as model swaps, prompt modifications, and tool updates.

• Develop evaluation systems utilizing golden datasets, LLM-as-judge workflows, and human review queues for both production and pre-production phases.

• Support new workflows during their incubation to ensure they are launched with appropriate instrumentation, evaluation coverage, and safety models.

• Convert technical changes into understandable language for leadership, clarifying what went wrong, its significance, and who is addressing the issue.

⛳️ Requirements

• Extensive production experience with LLMs, RAG, voice AI, agentic orchestration, or similar ML systems.

• Proven track record in designing or managing rubrics, scoring frameworks, regression tests, and production-sampling workflows.

• A well-defined mental framework for identifying various failure modes, including model reasoning, tool failures, retrieval challenges, and workflow logic.

• Comfortable with active experimentation—such as prompt/context tuning, retrieval redesign, and safe model migrations—beyond merely triaging issues.

• Strong cross-functional judgment and the ability to communicate effectively in ambiguous, high-accountability settings.

• A high degree of personal accountability with a commitment to owning results rather than merely offering recommendations.

• Experience in healthcare, regulated workflows, contact center operations, or high-stakes service quality and reliability functions.

🏝️ Benefits

• Employer-sponsored health, dental, and vision plans with low or no premiums.

• Generous paid time off.

• $100 monthly stipend for mobile or internet expenses.

• Stock options available for all employees.

• Bonus eligibility for all positions excluding Director and above; commission eligibility for Sales roles.

• Parental leave program.

• 401K program.

• And more....