Product Data Scientist – AI Evaluation, Quality

This is a fully remote position, open to applicants in Spain.

📋 Description

• Become a part of the AI Team leading all AI innovations and technologies at Finom.

• Take ownership of the offline evaluation suite across various products, encompassing capability and regression datasets, judges, and metrics.

• Develop and sustain online quality dashboards that track resolution rates, CSAT, thumbs up/down, LLM-as-judge signals, error rates, and latency.

• Close the feedback loop in production by analyzing failure patterns from actual traffic, converting them into regression cases, and suggesting solutions to Product and domain experts.

• Strengthen the evaluation methodology, focusing on judge stability and managing non-determinism.

• Convert metrics into actionable decisions through weekly meetings and clear trade-offs.

• Oversee the evaluation process across AI products, which includes pre-launch quality gates, post-launch monitoring, and ongoing improvements.

• Collaborate directly with the AI Quality lead, Igor Kolodkin.

• Work closely with AI engineers, Product, and domain specialists throughout the organization.

• Utilize tools such as Databricks, DeepEval, and Claude Code.

⛳️ Requirements

• Proficiency in Python and SQL to conduct comprehensive analysis from start to finish.

• Strong understanding of statistics, including sampling, hypothesis testing, variance, and handling noisy metrics.

• An analytical mindset that begins with the business question rather than focusing solely on the tools.

• Minimum of 3 years of experience in analyst or data scientist roles.

• At least 1 year of experience in a product-focused context.

• Experience in quality analytics for ML systems is advantageous.

• Practical experience in evaluating LLM applications (RAG, agents, tool use, judges) is a plus.

• Background in building LLM agents is preferred.

• Familiarity with AI-assisted coding as the primary authoring environment.

• Claude Code serves as the main tool for SQL, Python, analyses, dashboards, and internal scripts.

• Must be curious and proficient with AI coding or genuinely eager to quickly gain fluency.

🏝️ Benefits

• Ongoing personal and professional development resources and opportunities.

• Flexible travel and the option for remote or hybrid work throughout Europe.

• Stock Options Program accessible to every team member.

• A modern, friendly, and eco-conscious corporate culture.

• Consistent support and care for all employees.

• One month in a corporate apartment in Cyprus through the Work & Swim Program.