Senior AI Product Analyst – Automation Team

This is a fully remote position, open to applicants in Serbia.

📋 Description

• Take ownership of the complete product analytics framework for both AI-driven and traditional product features.

• Perform discovery and sizing, design and analyze experiments, conduct funnel and adoption analysis, and evaluate post-release impacts.

• Examine the operational behavior of AI and automation workflows throughout clinical and operational processes.

• Investigate issues such as hallucinations, silent failures, and declines in quality of AI-generated outputs.

• Analyze logs, transcripts, generated artifacts, and operational data to uncover patterns that may be overlooked by dashboards.

• Develop both automated and manual review assessments to evaluate AI quality.

• Integrate evaluation findings to inform prompt and product modifications.

• Create monitoring systems and dashboards to track AI and automation health along with relevant operational metrics.

• Assess the business and operational impacts of automation projects and highlight associated risks.

• Collaborate with Product, Engineering, Clinical, MedOps, and Legal teams to facilitate swift decision-making in uncertain situations.

⛳️ Requirements

• Over 5 years of experience in Product Analytics, Operational Analytics, Data Analytics, or similar analytics roles within intricate production frameworks.

• Solid understanding of AI, ML, and LLM systems.

• Practical experience in evaluating LLM or AI outputs is a significant advantage, including experience with prompt iteration, evaluation frameworks, and human review processes.

• Strong, organized written and verbal communication skills.

• Capability to transform analyses into concise, actionable documents for non-analysts.

• Ability to thrive in ambiguous and rapidly changing environments where consensus on ground truth must be established.

• Robust background in product and operational analytics.

• Experience managing metrics from start to finish.

• Proficiency in writing SQL and Python code to work with complex production data.

• Fluent in Russian.

• Proficient in English at B2 level or higher.

• Availability to work during CET-overlapping hours in a B2B context.

• Experience in evaluating LLM outputs or AI systems is a plus.

• Prior experience in Data Science or Machine Learning is advantageous.

🏝️ Benefits

• Competitive salary with clear opportunities for advancement.

• Fully remote and long-term B2B collaboration.

• Health insurance provided after the probation period.

• Compensation for sports and wellness activities.

• Opportunities for career development.

• Personalized English lessons through Preply.

• 19 working days of vacation annually.

• 4 additional wellness days.

• Paid sick leave for the first 5 working days.

• Thoughtful gifts for significant life events.