This is a fully remote position, open to applicants in California.

📋 Description

• The primary focus of this role is to evaluate agents powered by LLM technology.

• Design, implement, and operate evaluation frameworks that utilize LLMs as judges.

• Assess the quality, accuracy, relevance, and reliability of agent responses.

• Work in partnership with Business Analysts and stakeholders to create, validate, and maintain golden datasets.

• Instrument AI agents with the Langfuse platform.

• Trace agent executions, capture evaluation scores, monitor performance, and troubleshoot any failures.

• Conduct tests on agent behavior, tool utilization, reasoning workflows, and response quality.

⛳️ Requirements

• A minimum of 14 years of relevant experience.

• Practical experience in designing, implementing, and managing evaluation frameworks based on LLM technology to assess the quality, accuracy, relevance, and reliability of agent responses.

• Proven track record of collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets for agent evaluation purposes.

• Hands-on experience with Langfuse for instrumenting AI agents, which includes tracing agent execution, capturing evaluation scores, monitoring performance, and troubleshooting issues.

• A strong grasp of comprehensive agent evaluation processes, including testing agent behavior, tool usage, reasoning workflows, and response quality.

• Actual experience in implementing agent evaluation frameworks; having only theoretical knowledge or general GenAI/LLM development experience is not sufficient.