Product Data Scientist – AI Evaluation, Quality

This is a fully remote position, open to applicants in Germany.

📋 Description

• Take ownership and enhance our offline evaluation suite across various products — including datasets (capability + regression), judges, and metrics.

• Develop and manage online quality dashboards: resolution rate, customer satisfaction (CSAT), thumbs up/down, LLM-as-judge signals, error rate, and latency.

• Complete the production feedback loop: analyze failure patterns from actual traffic → convert them into regression cases → suggest solutions to Product and domain experts.

• Strengthen methodology: assess judge stability and manage non-determinism.

• Convert data into actionable decisions – conduct weekly syncs, clarify trade-offs, and avoid creating dashboards without purpose.

⛳️ Requirements

• Minimum of 3 years of experience in analyst or data scientist positions, with at least one year in a product environment.

• Proficiency in Python and SQL — capable of executing an analysis from start to finish.

• Strong grounding in statistics — including sampling, hypothesis testing, and variance analysis.

• Analytical mindset — approach problems starting from the business question, rather than the tools available.

• Experience in quality analytics for machine learning systems — including ranking, recommendations, and classification (Nice-to-Haves).

• Practical experience in evaluating LLM applications (RAG, agents, tool usage, judges) (Nice-to-Haves).

• Experience in building LLM agents — whether through side projects, small builds, or personal experiments is valuable (Nice-to-Haves).

• We seek analysts who are naturally curious and familiar with AI coding — or genuinely enthusiastic about becoming proficient quickly.

🏝️ Benefits

• Make a real difference in the product.

• Be part of our upward trajectory and grow alongside us.

• We offer resources and opportunities for ongoing personal and professional growth.

• Enjoy the flexibility of working remotely or in a hybrid model across Europe.

• Become a holder of stock options.

• Unleash your entrepreneurial spirit through our Stock Options Program.

• Receive unwavering support and care from our team.

• We provide continuous assistance to ensure your Finom experience is rewarding and successful.

• Participate in our exclusive Work & Swim Program.

• Spend a month in a comfortable corporate apartment in Cyprus.

• This is the perfect opportunity to achieve an ideal work-life balance.