Senior AI Product Manager – AI Builder

This is a fully remote position, open to applicants in California, +6 more states.

📋 Description

• Collaborate with operators, customers, and domain experts within Mobility, Payments, and Benefits to identify workflows, friction points, and failure demand.

• Establish future-state experiences and baseline performance metrics such as cycle time, manual hours, containment, and error rates.

• Identify high-value workflow segments to assess feasibility and build trust.

• Quickly develop and evaluate agent behavior, system context, prompts, tool flows, retrieval/grounding, and orchestrations involving multi-agent or human-in-the-loop interactions.

• Work alongside the Tech Builder to modify repositories and configurations, test scenarios, analyze execution traces, and troubleshoot agent logic.

• Differentiate between exploratory learning prototypes and production systems that necessitate engineering hardening.

• Design representative, edge-case, and adversarial AI evaluation scenarios.

• Set pass/fail criteria, assertions, and scoring rubrics for accuracy, tool selection, citation quality, policy compliance, hallucination rates, as well as PII/PHI/PCI safety, and cost per outcome.

• Investigate failed traces and create error taxonomies covering prompt, context, retrieval, model, and UX issues.

• Convert insights into versioned repository artifacts, including PRDs, acceptance criteria, decision logic, evaluation harnesses, and production metrics.

• Ensure traceability from user problems through workflows, requirements, scenarios, evaluations, and production metrics.

• Navigate legacy systems, fragmented APIs, and intricate operational policies to showcase functioning software in real-world settings.

• Collaborate with Risk, Compliance, Security, Legal, and AI Governance on decision boundaries, entitlements, auditability, prompt-injection defenses, and fallback controls.

• Instrument live product usage to link agent performance with business ROI.

• Develop reusable patterns, runbooks, and controls for domain teams to manage and enhance mature solutions.

⛳️ Requirements

• Over 7 years of Product Management experience delivering software, data, platform, automation, or AI products to production at scale.

• Direct experience with generative AI, LLMs, retrieval/grounding (RAG), tool calling, context management, orchestration, structured outputs, and human-in-the-loop workflows.

• Proven experience in constructing or managing AI evaluations using golden datasets, quality rubrics, failure analysis, or model-based judges.

• Proficient in utilizing AI-assisted development tools, low-code frameworks, APIs, and version-controlled Markdown, JSON, and YAML configuration files.

• Capability to balance accuracy, autonomy, latency, token/system cost, security, and user experience.

• Insight into when to apply probabilistic AI versus deterministic code.

• Demonstrated success in fostering alignment among Engineers, Risk, Legal, and Operations teams.

• Bachelor’s degree in a relevant field or equivalent practical experience.

• Preferred: experience in regulated sectors such as Fintech, Payments, Health/Benefits, Fleet, or Financial Services.

• Preferred: familiarity with enterprise AI governance frameworks, model risk management, responsible AI, or auditability.

• Preferred: hands-on experience with agent frameworks, prompt/context engineering tools, or AI experimentation platforms.

🏝️ Benefits

• Health, dental, and vision insurance.

• Retirement savings plan.

• Paid time off.

• Health savings account.

• Flexible spending accounts.

• Life insurance.

• Disability insurance.

• Tuition reimbursement.

• Eligibility for quarterly or annual bonuses for non-sales roles.

• Comprehensive and competitive benefits package.

• Total compensation package.