Senior AI Product Analyst – Automation Team

This is a fully remote position, open to applicants in Spain.

📋 Description

• Take ownership of the complete product analytics stack for both AI-driven and traditional product features.

• Perform discovery and sizing, design and analyze experiments, conduct funnel and adoption analysis, and assess post-release impact.

• Examine production behavior for AI and automation workflows throughout clinical and operational processes.

• Investigate issues such as hallucinations, silent failures, and quality degradation in AI-generated outputs.

• Analyze logs, transcripts, generated artifacts, and operational data to uncover patterns that dashboards may overlook.

• Create evaluations for AI quality that incorporate both automated and human review processes.

• Integrate evaluation insights into prompts and product modifications.

• Develop monitoring systems and dashboards for tracking the health of AI and automation, along with related operational metrics.

• Assess the business and operational impact of automation initiatives while identifying associated risks.

• Collaborate with Product, Engineering, Clinical, MedOps, and Legal teams to inform decisions amidst uncertainty.

⛳️ Requirements

• Over 5 years of experience in Product Analytics, Operational Analytics, Data Analytics, or similar roles in complex production environments.

• Knowledge of AI, ML, and LLM systems.

• Practical experience in evaluating LLM or AI outputs; expertise in prompt iteration, evaluation frameworks, and human review pipelines is highly advantageous.

• Strong written and verbal communication skills that are effective and structured.

• Capability to convert analysis into clear, actionable documents that non-analysts can utilize.

• Comfort in fast-paced, ambiguous settings where establishing the ground truth is often necessary.

• A solid background in product or operational analytics.

• Experience managing metrics comprehensively from start to finish.

• Proficiency in writing SQL and Python code to handle complex production data.

• Fluent in Russian.

• Proficient in English at a B2 level or higher.

• Willingness to work during CET-overlapping hours in a B2B context.

🏝️ Benefits

• Competitive salary with clear opportunities for advancement.

• Fully remote position with long-term B2B collaboration.

• Health insurance provided after the probation period.

• Compensation for sports and wellness activities.

• Career development opportunities, including personalized English lessons through Preply.

• 19 working days of vacation annually.

• 4 additional wellness days.

• Paid sick leave for the first 5 working days.

• Thoughtful gifts for significant life events.