Product Data Scientist – AI Evaluation, Quality

This is a fully remote position, open to applicants in Lithuania.

📋 Description

• Take ownership of and enhance our offline evaluation suite across various products, including datasets (capability + regression), judges, and metrics.

• Develop and sustain online quality dashboards that track resolution rates, CSAT, thumbs up/down feedback, LLM-as-judge signals, error rates, and latency.

• Close the production feedback loop by analyzing failure patterns from real traffic and suggesting improvements.

• Strengthen our methodology by ensuring judge stability and effective handling of non-determinism.

• Convert numerical insights into actionable decisions during weekly syncs, highlighting clear trade-offs.

⛳️ Requirements

• Proficiency in Python and SQL, enabling you to conduct analysis from start to finish.

• Strong background in statistics, including sampling, hypothesis testing, variance, and understanding noisy metrics.

• An analytical mindset that begins with the business question rather than the tool.

• A minimum of 3 years of experience in analyst or data scientist positions, with at least one role within a product context.

• Experience in quality analytics for machine learning systems is advantageous.

• Practical experience in evaluating LLM applications is a plus.

• Experience in building LLM agents is a plus.

🏝️ Benefits

• Make a significant impact on the product.

• Join us on our upward trajectory and grow alongside the company.

• Access resources and opportunities for ongoing personal and professional development.

• Enjoy the flexibility of traveling and working remotely or in a hybrid model across Europe.

• Become a stock options holder through our Stock Options Program.

• Experience unwavering support and care from Finom.

• Participate in the exclusive Work & Swim Program.

• Engage in a modern, friendly, and eco-conscious corporate culture.

• Work in an equal opportunity environment that values diversity.