Senior Applied AI Engineer, Agent Quality & Evaluations

This is a fully remote position, open to applicants in California.

📋 Description

• Oversee member-facing AI behavior throughout the entire process from prompt creation to agentic editing, as well as future tasks such as segmentation, scheduling, analytics, and recommendations.

• Develop, test, version, and document prompts and agent instructions.

• Determine the system’s use of context, member data, tools, and product status.

• Construct and prototype agent loops, tools, and structured interfaces.

• Lead AI evaluation practices, which includes representative datasets, behavioral scenarios, scoring rubrics, regression suites, and human review.

• Build and manage evaluation harnesses, monitoring systems, and quality dashboards.

• Identify failures across prompts, context, orchestration, models, tools, data, and product code.

• Establish quality baselines and release gates for changes to prompts, models, and agents.

• Transform production failures, traces, and member feedback into sustainable regression cases.

• Collaborate with product, design, marketing, and copy specialists to integrate Flodesk's perspective into the system.

⛳️ Requirements

• Proven experience in building or significantly enhancing production-level LLM or agentic products, beyond mere prototypes.

• Proficient software engineering capabilities in Python, TypeScript, or a comparable language.

• Comprehension of the interplay between prompts, context, tools, agent loops, orchestration, and product state.

• Direct involvement with LLM evaluation, observability, and experimental design.

• Capacity to merge automated checks with human insight.

• Strong product intuition for converting ambiguous member feedback into testable quality definitions and specific system modifications.

• Systems thinking and the ability to recognize reusable patterns.

• Effective communication skills across both technical and non-technical fields.

• Bonus: experience in developing content-generation, creative, or marketing tools.

• Bonus: experience constructing multi-turn agents that utilize tools while maintaining user intent throughout edits.

• Bonus: familiarity with Langfuse, Braintrust, or similar evaluation and observability platforms.

• Bonus: experience working on products designed for small businesses, creators, or marketers.

🏝️ Benefits

• Comprehensive health insurance coverage for individuals at no cost.

• 16 weeks of paid parental leave for non-birthing parents.

• 22 weeks of paid maternity leave for birthing parents.

• Unlimited flexible time off.

• 401(k) matching for US employees.

• $1,000 annual stipend dedicated to learning and development.