AI Product Analyst – Automation Team

This is a fully remote position, open to applicants in Georgia.

📋 Description

• Take ownership of the complete product analytics stack for features delivered by the Automation team, encompassing discovery and sizing, experiment design and analysis, funnel and adoption evaluation, and post-release impact assessment.

• Evaluate the operational behavior of AI and automation workflows throughout clinical and operational journeys.

• Examine hallucinations, silent failures, and quality declines in AI-generated outputs.

• Collaborate directly with logs, transcripts, generated artifacts, and operational data to uncover patterns that dashboards may overlook.

• Create automated and human-review assessments for AI quality and integrate findings into prompt and product modifications.

• Develop monitoring systems and dashboards for AI and automation health, along with relevant operational metrics.

• Assess the business and operational impact of automation projects and highlight associated risks.

• Collaborate with Product, Engineering, Clinical, MedOps, and Legal teams to facilitate quick decision-making amidst uncertainty.

⛳️ Requirements

• Over 3 years of experience in Product Analytics, Operational Analytics, Data Analytics, or similar analytics roles within complex production systems.

• Knowledge of AI, ML, and LLM systems.

• Practical experience in evaluating LLM or AI output is highly advantageous.

• Familiarity with prompt iteration, evaluation frameworks, and human review processes is a significant plus.

• Strong, structured written and verbal communication skills.

• Ability to convert analysis into concise, decision-ready documents that non-analysts can utilize.

• Comfortable working in ambiguous, rapidly changing environments where ground truth often needs to be established.

• Robust background in product and operational analytics.

• Capability to manage metrics from start to finish.

• Proficient in writing SQL and Python code to work with complex production data.

• Fluent in Russian.

• Proficient in English at a B2 level or higher.

• Willingness to work during CET-overlapping hours in a B2B setting.

• Experience in evaluating LLM outputs or AI systems is a plus.

• Previous background in Data Science or Machine Learning is also a plus.

🏝️ Benefits

• Competitive salary and opportunities for professional growth.

• Health insurance available after the probationary period.

• Compensation for sports and wellness activities.

• Fully remote position with long-term B2B collaboration.

• 19 working days of vacation annually.

• 3 additional wellness days per year.

• Paid sick leave for the first 5 working days.

• Gifts for employees on significant occasions.

• Personalized English lessons provided through Preply.