Senior AI Product Analyst – Voice AI Agents

This is a fully remote position, open to applicants in Serbia.

📋 Description

• Assess the production behavior of Voice AI Agents and conversational AI workflows.

• Examine hallucinations, inconsistencies, silent failures, and quality declines in AI-driven dialogues.

• Collaborate with call transcripts, logs, generated outputs, and operational datasets to identify patterns and actionable insights.

• Create evaluation frameworks to assess AI quality, reliability, and business impact.

• Develop monitoring systems and dashboards to enhance AI performance visibility.

• Evaluate the effect of AI systems on operational efficiency, business outcomes, and patient experience.

• Identify quality shifts, anomalies, and emerging failure modes in production settings.

• Collaborate with Product, Engineering, QA, and Operations teams on AI projects.

• Facilitate quick decision-making in rapidly changing and uncertain environments.

⛳️ Requirements

• Over 5 years of experience in Product Analytics, Data Analytics, Operational Analytics, or related analytical roles within complex production systems.

• Proficient in SQL.

• Experience using Python for analytical research and handling large-scale text datasets.

• Familiarity with ML pipelines or AI-driven products.

• Knowledge of LLM limitations, hallucinations, evaluation challenges, prompt sensitivity, and AI failure modes.

• Experience in creating monitoring systems, analytical frameworks, or operational dashboards.

• Excellent communication skills with the ability to convert analysis into product decisions.

• Capability to work efficiently in ambiguous and fast-paced environments.

• Fluent in Russian.

• English proficiency at B2 level or higher.

• Willingness to work during CET-overlapping hours.

• Background in conversational AI or AI agents.

• Experience in evaluating LLM outputs.

• Familiarity with AI quality metrics and evaluation methodologies.

• Experience with prompt iteration and AI evaluation frameworks.

🏝️ Benefits

• Competitive salary and clear opportunities for advancement.

• Fully remote, long-term B2B collaboration.

• Health insurance provided after the probation period.

• Compensation for sports and wellness activities.

• Career development opportunities.

• Personalized English lessons through Preply.

• 19 working days of vacation annually.

• 4 additional wellness days.

• Paid sick leave for the first 5 working days.

• Thoughtful gifts for significant life events.