AI Product Analyst – Automation Team

This is a fully remote position, open to applicants in Serbia.

📋 Description

• Take full ownership of the entire product analytics framework for features developed by the team, encompassing discovery and sizing, experiment design and analysis, funnel and adoption analysis, and post-release impact assessment.

• Examine the production behavior of AI and automation workflows throughout clinical and operational processes.

• Explore issues such as hallucinations, silent failures, and quality degradation in outputs generated by AI.

• Utilize logs, transcripts, generated artifacts, and operational data to uncover patterns that dashboards may overlook.

• Create both automated and human-review evaluations to assess AI quality, integrating findings into adjustments in prompts and product changes.

• Develop monitoring systems and dashboards to track AI and automation health along with related operational metrics.

• Assess the business and operational impacts of automation initiatives while identifying associated risks.

• Collaborate with Product, Engineering, Clinical, MedOps, and Legal teams to facilitate rapid decision-making in uncertain situations.

⛳️ Requirements

• Minimum of 3 years of experience in Product Analytics, Operational Analytics, Data Analytics, or similar analytical roles within complex production environments.

• Familiarity with AI, ML, and LLM systems.

• Practical experience in evaluating LLM or AI outputs; proficiency in prompt iteration, evaluation frameworks, and human review pipelines is highly advantageous.

• Strong, structured skills in written and verbal communication.

• Capability to translate analysis into clear, actionable write-ups that can be utilized by non-analysts.

• Adaptability to ambiguous and rapidly evolving environments where establishing ground truth is often necessary.

• Solid background in product and operational analytics.

• Proficient in managing metrics from start to finish and skilled in writing SQL and Python code to handle complex production data.

• Fluent in Russian.

• Proficient in English at a B2 level or higher.

• Availability to work during CET-overlapping hours in a B2B context.

🏝️ Benefits

• Competitive salary and opportunities for professional growth.

• Health insurance provided after the probationary period.

• Compensation for sports and wellness activities.

• Fully remote, long-term collaboration in a B2B format.

• 19 working days of vacation each year.

• 3 additional wellness days annually.

• Paid sick leave for the first 5 working days.

• Gifts for employees on significant occasions.

• Customized English lessons through Preply.