Product Manager, Agentic AI

This is a fully remote position, open to applicants in United States.

📋 Description

• Take charge of the roadmap for Kinetic's agent orchestration layer, encompassing models and routing, tools and memory, observability, permissions, and guardrails.

• Manage the agent registry with clear documentation regarding ownership, purpose, autonomy tier, and behavior logging for each agent.

• Establish protocols for agent deployment, versioning, and rollback processes.

• Oversee management of prompts, tool definitions, and model versions as if they were code.

• Monitor usage, costs, and latency associated with each agent.

• Lead the initiative to clean up existing AI scripts and workflows by either retiring, replatforming, or sunsetting them.

• Ensure that business logic, specifications, and evaluation data are transferable across different frameworks and models.

• Collaborate with experience-team PMs to assess whether agents or simpler automation solutions should address business challenges.

• Convert use cases into agent specifications, detailing inputs, tools, autonomy boundaries, human escalation pathways, success metrics, and baselines.

• Author, test, and refine system prompts, tool descriptions, and structured outputs in partnership with agent and data engineers.

• Link each agent to a specific business metric prior to scaling efforts.

• Create golden datasets and execute regression suites whenever prompts, tools, or models undergo changes.

• Oversee rollout procedures from shadow mode to staged rollout and full production.

• Uphold the autonomy-tier policy and outline necessary human checkpoints.

• Generate architecture maps, evaluation histories, audit trails, and before-and-after metrics.

• Report directly to the VP of Product and collaborate with product managers across Broker, Policyholder, Underwriter, Claims, Data, and Business Systems teams.

⛳️ Requirements

• 5+ years of experience in Product Management, Technical Product Management, or a related field.

• At least 1 year of experience deploying LLM-powered or agentic products into production, beyond just demonstrations.

• Proficient in hands-on prompt engineering, including system prompts, tool definitions, multi-step chains, and structured outputs.

• Experience in constructing or conducting evaluations for LLM systems, such as creating golden datasets, LLM-as-judge graders, and regression testing.

• Familiarity with agent frameworks and platforms such as CrewAI, LangGraph, OpenAI Agents SDK, or AWS Bedrock AgentCore.

• Understanding of RAG, tool calling, and MCP.

• Capability to translate complex business requirements into clear specifications, user stories, and acceptance criteria.

• Background in startups or high-growth settings.

• Exceptional communication skills with the ability to articulate probabilistic systems to business leaders.

• Candidates must reside within the United States.

• Must have existing authorization to work in the United States.

• Verification of government-issued identity and employment eligibility will be required prior to employment.

• Previous experience in an insurtech, MGA, carrier, or another regulated sector such as financial services or healthcare is advantageous.

• Knowledge of workers' compensation, commercial lines underwriting, or claims workflows is a plus.

• Familiarity with AI governance frameworks is beneficial.

• Experience in multi-model routing, cost optimization, or human-in-the-loop design for agents is a plus.

🏝️ Benefits

• Fully distributed, remote team operating across all continental US time zones.

• Comprehensive medical, dental, and vision insurance coverage.

• 20 vacation days each year.

• 9 federal holidays off.

• Parental Leave.

• Pre-tax 401k contribution options.

• HSA with compatible health plans.

• FREE: Health Advocate services.

• FREE: Telehealth Membership.

• FREE: OneMedical Account.

• Domestic travel for company offsites and team alignment.