AI Product Analyst – Voice AI Agents

This is a fully remote position, open to applicants in Poland.

📋 Description

• Evaluate the performance of Voice AI Agents and conversational AI processes.

• Examine issues such as hallucinations, inconsistencies, silent failures, and quality decline in AI-driven dialogues.

• Review call transcripts, logs, generated outputs, and operational data to identify trends and actionable insights.

• Develop assessment frameworks for AI quality, reliability, and business impact.

• Create monitoring systems and dashboards to enhance visibility of AI performance.

• Assess the effect of AI on operational efficiency, business results, and patient experience.

• Identify quality changes, anomalies, and potential failure modes in production settings.

• Collaborate with Product, Engineering, QA, and Operations teams on AI-related projects.

• Facilitate quick decision-making in dynamic and uncertain environments.

⛳️ Requirements

• Over 3 years of experience in Product Analytics, Data Analytics, Operational Analytics, or similar analytics roles in complex production systems.

• Proficiency in SQL.

• Experience with Python for analytical tasks and handling large-scale text datasets.

• Awareness of LLM limitations, hallucinations, evaluation hurdles, prompt sensitivity, and AI failure modes.

• Background in creating monitoring systems, analytical frameworks, or operational dashboards.

• Excellent communication skills with the ability to convert analysis into product strategies.

• Capacity to thrive in ambiguous and rapidly changing environments.

• Fluent in Russian.

• English proficiency at B2 level or above.

• Availability to work during CET-overlapping hours.

• Experience with conversational AI or AI agents is a plus.

• Familiarity with evaluating LLM outputs is a plus.

• Knowledge of AI quality metrics and evaluation methods is a plus.

• Experience with prompt iteration and AI evaluation frameworks is a plus.

• Background working with ML pipelines or AI-powered products is a plus.

🏝️ Benefits

• Competitive salary with clear opportunities for advancement.

• Fully remote, long-term B2B collaboration.

• Health insurance provided after the probation period.

• Compensation for sports and wellness activities.

• Opportunities for career development.

• Customized English lessons through Preply.

• 19 days of paid vacation annually.

• 3 additional wellness days.

• Paid sick leave for the first 5 working days.

• Thoughtful gifts for significant life events.