Lead Product Manager, Agent Runtime, Evals

This is a fully remote position, open to applicants in United States, +1 more country.

📋 Description

• Take ownership of the vision, strategy, and roadmap for Foundry's agent runtime and evaluation layer.

• Establish the lifecycle for agents, applications, and scripts: draft, test, publish, version, deploy, run, observe, enhance, and retire.

• Develop evaluation processes utilizing golden datasets, data labeling, regression gates, automated judges, and human review.

• Determine model eligibility, routing, cost attribution, and pricing for automations.

• Define agent memory, learning capabilities, self-improvement, identity, access grants, and operational modes.

• Integrate hosted agents with Juniper Square products, fund administration systems, and third-party applications.

• Design oversight tools such as dashboards, run history, traces, metrics, logs, kill switches, and audit capabilities.

• Collaborate with the workflow orchestration PM on agent registries, typed contracts, and workflow bindings.

• Create configuration tools for business SMEs to set up, test, and optimize automations.

• Monitor the managed agent runtime, framework, and automation-platform market and make decisions to build, buy, or partner.

• Lead across platform and related teams, navigating ambiguous platform trade-offs.

• Report to the Group Product Manager for Foundry while working alongside a dedicated engineering team.

• Implement real fund administration workflows and automations on Foundry, including capital calls and distributions.

⛳️ Requirements

• Over 7 years of product management experience in B2B or Enterprise SaaS, with at least 3 years in an AI-focused role.

• Demonstrated experience managing a complex product area from strategy and roadmap to execution and iteration, preferably in a zero-to-one environment.

• Practical knowledge of agent evaluation, encompassing golden datasets, data labeling, automated judges, regression testing, and human-in-the-loop decision-making.

• Background in developing agents and transitioning human-driven processes to products and automation, with a focus on real evaluation and monitoring in production.

• Strong engineering collaboration skills and understanding of typed contracts between agents and workflows, as well as structured and deterministic outputs, and agent memory.

• User empathy for non-technical builders and operators managing numerous automated agents.

• Proficiency with LLMs and popular AI tools such as Claude and Cursor.

• Capability to run prototypes and leverage AI tools to expedite product development.

• Excellent written and verbal communication abilities.

• Proven track record of leading through influence, ideally mentoring or managing other product managers, and establishing strategy across multiple teams.

🏝️ Benefits

• Health, dental, and vision insurance for you and your family.

• Life insurance coverage.

• Support for mental wellness.

• Assistance for fertility and growing family needs.

• Flex Time Off in addition to company-paid holidays.

• Paid family leave, medical leave, and bereavement leave policies.

• Retirement savings plans.

• Allowance for customizing your work and technology setup at home.

• Annual professional development stipend.

• Equity opportunities.