AI Product Analyst – Automation Team

This is a fully remote position, open to applicants in Spain.

📋 Description

• Take full ownership of the product analytics stack for team-released features, encompassing discovery and sizing, experiment design and analysis, funnel and adoption analysis, along with post-release impact evaluation.

• Examine the production behavior of AI and automation workflows throughout clinical and operational processes.

• Explore hallucinations, silent failures, and quality deterioration in outputs generated by AI.

• Work directly with logs, transcripts, generated artifacts, and operational data to uncover patterns that dashboards may overlook.

• Create evaluations for AI quality that incorporate both automated and human reviews, using findings to inform prompt and product modifications.

• Develop monitoring systems and dashboards for assessing AI and automation health along with relevant operational metrics.

• Assess the business and operational impact of automation initiatives while identifying associated risks.

• Collaborate with Product, Engineering, Clinical, MedOps, and Legal teams to facilitate swift decision-making in uncertain situations.

⛳️ Requirements

• At least 3 years of experience in Product Analytics, Operational Analytics, Data Analytics, or similar analytics roles within complex production environments.

• Familiarity with AI, machine learning (ML), and large language model (LLM) systems.

• Practical experience in evaluating LLM or AI outputs, including prompt iteration, evaluation frameworks, and human review processes is highly advantageous.

• Strong written and verbal communication skills with an effective and structured approach.

• Ability to thrive in fast-paced, ambiguous environments where establishing ground truth is often necessary.

• A solid background in product and operational analytics.

• Proven experience in managing metrics from start to finish.

• Proficiency in writing SQL and Python code to analyze complex production data.

• Fluent in Russian.

• English proficiency at B2 level or higher.

• Capability to work during CET-overlapping hours in a B2B context.

• Experience in evaluating LLM outputs or AI systems is a plus.

• Previous background in Data Science or Machine Learning is also a plus.

🏝️ Benefits

• Health insurance provided after the probation period.

• Compensation for sports and wellness activities.

• Fully remote, long-term B2B collaboration opportunities.

• 19 working days of vacation annually.

• 3 additional wellness days each year.

• Paid sick leave for the first 5 working days.

• Gifts for employees during significant events.

• Personalized English lessons available via Preply.

• Competitive salary and opportunities for professional growth.