AI Product Analyst – Voice AI Agents

This is a fully remote position, open to applicants in Georgia.

📋 Description

• Examine the production behavior of Voice AI Agents and conversational AI workflows.

• Investigate hallucinations, inconsistencies, silent failures, and quality deterioration in AI-driven conversations.

• Analyze call transcripts, logs, generated outputs, and operational datasets to identify patterns and derive actionable insights.

• Develop evaluation frameworks to assess AI quality, reliability, and business impact.

• Create monitoring systems and dashboards for enhanced visibility into AI performance.

• Assess the influence of AI systems on operational efficiency, business outcomes, and patient experience.

• Identify quality shifts, anomalies, and new failure modes within production environments.

• Collaborate with Product, Engineering, QA, and Operations teams on AI initiatives.

• Facilitate rapid decision-making in dynamic and ambiguous settings.

⛳️ Requirements

• A minimum of 3 years of experience in Product Analytics, Data Analytics, Operational Analytics, or similar analytics roles within complex production systems.

• Proficient in SQL.

• Experience with Python for analytical investigations and handling large-scale text datasets.

• Knowledge of LLM limitations, hallucinations, evaluation challenges, prompt sensitivity, and AI failure modes.

• Proven experience in building monitoring systems, analytical frameworks, or operational dashboards.

• Excellent communication skills with the ability to translate analysis into product decisions.

• Capable of working effectively in ambiguous and rapidly changing environments.

• Fluent in Russian.

• Proficient in English at level B2 or higher.

• Availability to work during CET-overlapping hours.

• Experience with conversational AI or AI agents is a plus.

• Familiarity with evaluating LLM outputs is a plus.

• Knowledge of AI quality metrics and evaluation methodologies is a plus.

• Experience with prompt iteration and AI evaluation frameworks is a plus.

• Background in working with ML pipelines or AI-powered products is a plus.

🏝️ Benefits

• Competitive salary and clear opportunities for growth.

• Health insurance provided after the probation period.

• Compensation for sports and wellness activities.

• Opportunities for career advancement.

• Personalized English lessons through Preply.

• 19 working days of vacation annually.

• 3 additional wellness days.

• Paid sick leave for the first 5 working days.

• Thoughtful gifts for significant life events.

• Fully remote position with long-term B2B collaboration.