AI Product Analyst – Automation Team

This is a fully remote position, open to applicants in Poland.

📋 Description

• Take ownership of the entire product analytics framework for the features delivered by the team, encompassing discovery and sizing, experiment design and analysis, funnel and adoption assessment, and post-release impact evaluation.

• Examine the production behavior of AI and automation workflows throughout clinical and operational journeys.

• Investigate instances of hallucinations, silent failures, and quality decline in AI-generated outputs.

• Work directly with logs, transcripts, generated artifacts, and operational data to identify patterns that dashboards may overlook.

• Develop automated and human-review evaluation processes for assessing AI quality.

• Ensure a continuous feedback loop for evaluation is integrated into prompt and product modifications.

• Create monitoring systems and dashboards that track the health of AI and automation, as well as related operational metrics.

• Assess the business and operational impact of automation initiatives while highlighting associated risks.

• Collaborate with Product, Engineering, Clinical, MedOps, and Legal teams to facilitate swift decision-making in uncertain situations.

⛳️ Requirements

• A minimum of 3 years of experience in Product Analytics, Operational Analytics, Data Analytics, or related analytics roles within complex production environments.

• Familiarity with AI, ML, and LLM systems.

• Practical experience in evaluating LLM or AI outputs; proficiency in prompt iteration, evaluation frameworks, and human review processes is highly desirable.

• Strong structured written and verbal communication skills.

• Capability to transform analysis into clear, decision-ready documentation that non-analysts can utilize.

• Comfort in dynamic, ambiguous environments where it may be necessary to establish ground truth.

• A solid background in product and operational analytics.

• Experience in managing metrics from start to finish.

• Proficiency in writing SQL and Python code to handle complex production data.

• Fluent in Russian.

• Proficient in English at a B2 level or higher.

• Availability to work during CET-overlapping hours in a B2B context.

• Experience in evaluating LLM outputs or AI systems is a plus.

• Previous background in Data Science or Machine Learning is advantageous.

🏝️ Benefits

• Competitive compensation and opportunities for growth.

• Health insurance provided after the probation period.

• Compensation for sports and wellness activities.

• Fully remote, long-term B2B collaboration.

• 19 working days of vacation annually.

• 3 additional wellness days each year.

• Paid sick leave for the initial 5 working days.

• Gifts for employees during significant events.

• Personalized English lessons through Preply.