Product Data Scientist – AI Evaluation, Quality

This is a fully remote position, open to applicants in Poland.

📋 Description

• Take ownership of and enhance our offline evaluation suite across various products, including datasets (capability + regression), judges, and metrics.

• Develop and oversee online quality dashboards that track resolution rates, CSAT, thumbs up/down feedback, LLM-as-judge signals, error rates, and latency metrics.

• Close the production feedback loop by analyzing failure patterns from real traffic, converting them into regression cases, and suggesting solutions to Product and domain experts.

• Strengthen methodology by ensuring judge stability and effective handling of non-determinism.

• Convert data into actionable decisions through weekly meetings, clear trade-offs, and avoidance of unnecessary dashboards.

⛳️ Requirements

• Proficiency in Python and SQL, enabling you to conduct analyses from start to finish.

• Strong background in statistics, including sampling, hypothesis testing, variance, and an understanding of noisy metrics.

• An analytical approach that begins with business questions rather than tools.

• A minimum of 3 years in analyst or data scientist positions, with at least one role in a product context.

• Direct experience in evaluating LLM applications (RAG, agents, tool use, judges).

• Experience in developing LLM agents, whether through side projects, small builds, or personal experiments.

• Familiarity with AI-assisted coding as our standard authoring environment.

• A genuine curiosity about AI coding, or a strong desire to quickly become proficient.

• A commitment to the quality of your work and clarity of thought.

🏝️ Benefits

• Make a significant impact on the product.

• Join our upward trajectory and grow with us.

• Work within the EU.

• Enjoy the flexibility of traveling and working remotely or in a hybrid model across Europe.

• Become a holder of stock options.

• Unlock your entrepreneurial spirit through our Stock Options Program.

• Receive unwavering support and care to ensure your experience is successful and fulfilling.

• Immerse yourself in our exclusive Work & Swim Program.

• Spend a month in a comfortable corporate apartment in beautiful Cyprus.

• An ideal opportunity for achieving work-life balance while enjoying stunning Mediterranean views.

• We are an Equal Opportunity Employer that values diversity.