Senior AI Product Analyst – Automation Team

This is a fully remote position, open to applicants in Poland.

📋 Description

• Take ownership of the complete product analytics framework for both AI-powered and traditional product features.

• Execute discovery and sizing, experiment design and evaluation, funnel and adoption analysis, as well as post-release impact assessments.

• Examine the operational behavior of AI and automation workflows across clinical and operational processes.

• Explore hallucinations, silent failures, and quality declines in AI-generated outputs.

• Analyze logs, transcripts, generated artifacts, and operational data to detect patterns that dashboards may overlook.

• Create automated evaluations and human-review assessments for AI quality.

• Integrate evaluation findings into prompt adjustments and product modifications.

• Develop monitoring systems and dashboards for the health of AI and automation, along with relevant operational metrics.

• Assess the business and operational impact of automation projects and identify related risks.

• Collaborate with Product, Engineering, Clinical, MedOps, and Legal teams to facilitate quick decision-making in uncertain situations.

⛳️ Requirements

• Minimum of 5 years of experience in Product Analytics, Operational Analytics, Data Analytics, or similar analytics roles within intricate production systems.

• Familiarity with AI, ML, and LLM systems.

• Practical experience in evaluating LLM or AI outputs is highly advantageous, including prompt iteration, evaluation frameworks, and human review processes.

• Strong, structured written and verbal communication skills.

• Capability to transform analysis into concise, decision-ready documentation for non-analysts.

• Comfort in ambiguous and rapidly changing environments where the truth often needs to be established.

• Solid background in product and operational analytics.

• Experience managing metrics from start to finish.

• Proficiency in writing SQL and Python code for complex production data.

• Fluent in Russian.

• Proficient in English at a B2 level or higher.

• Availability to work during CET-overlapping hours in a B2B context.

• Experience evaluating LLM outputs or AI systems is a plus.

• Prior experience in Data Science or Machine Learning is advantageous.

🏝️ Benefits

• Competitive salary and clear opportunities for advancement.

• Fully remote, long-term B2B collaboration.

• Health insurance provided after the probation period.

• Compensation for sports and wellness activities.

• Opportunities for career development.

• Personalized English lessons through Preply.

• 19 working days of vacation annually.

• 4 additional wellness days.

• Paid sick leave for the first 5 working days.

• Thoughtful gifts for significant life events.