Senior AI Product Analyst – Automation Team

This is a fully remote position, open to applicants in Georgia.

📋 Description

• Take ownership of the entire product analytics framework for the features delivered by the team, encompassing discovery and sizing, experiment design and analysis, funnel and adoption analysis, as well as post-release impact evaluation.

• Analyze the operational behavior of AI and automation workflows throughout clinical and operational processes.

• Examine hallucinations, silent failures, and quality degradation in outputs generated by AI.

• Work directly with logs, transcripts, generated artifacts, and operational data to identify patterns that dashboards may overlook.

• Develop both automated and human-review evaluation methods to assess AI quality.

• Incorporate evaluation findings into prompt and product modifications.

• Create monitoring systems and dashboards to track the health of AI and automation along with relevant operational metrics.

• Evaluate the business and operational impact of automation projects.

• Highlight risks associated with automation initiatives.

• Collaborate with Product, Engineering, Clinical, MedOps, and Legal teams to facilitate swift decision-making in uncertain scenarios.

⛳️ Requirements

• A minimum of 5 years' experience in Product Analytics, Operational Analytics, Data Analytics, or similar analytics roles within intricate production systems.

• Familiarity with AI, ML, and LLM systems.

• Practical experience in evaluating LLM or AI outputs is highly desirable, including prompt iteration, evaluation frameworks, and human review processes.

• Strong, structured written and verbal communication skills.

• Capability to transform analysis into concise, decision-ready documents that non-analysts can utilize.

• Comfort in dynamic, ambiguous environments where the truth often needs to be established.

• Solid background in product and operational analytics.

• Proficient in managing metrics from start to finish.

• Ability to write SQL and Python code to handle complex production data.

• Proficiency in Russian.

• English proficiency at B2 level or higher.

• Availability to work during CET-overlapping hours in a B2B context.

• Experience in evaluating LLM outputs or AI systems is a plus.

• Prior experience in Data Science or Machine Learning is advantageous.

🏝️ Benefits

• Competitive salary and clear pathways for career advancement.

• Health insurance provided after the probation period.

• Compensation for sports and wellness activities.

• Tailored English lessons through Preply.

• 19 working days of vacation annually.

• 4 additional wellness days.

• Paid sick leave for the initial 5 working days.

• Thoughtful gifts for significant life events.

• Fully remote, long-term B2B engagement.