Product Data Scientist – AI Evaluation, Quality

This is a fully remote position, open to applicants in Portugal.

📋 Description

• Take ownership and enhance our offline evaluation suite across various products — including datasets (capability + regression), judges, and metrics.

• Develop and sustain online quality dashboards that track resolution rates, CSAT, thumbs up/down, LLM-as-judge signals, error rates, and latency.

• Close the feedback loop in production: analyze failure patterns from actual traffic → convert them into regression cases → suggest solutions to Product and domain experts.

• Strengthen methodology: ensuring judge stability and management of non-determinism.

• Convert data into actionable decisions – conduct weekly syncs, clarify trade-offs, and avoid creating dashboards that serve no purpose.

⛳️ Requirements

• Proficient in Python and SQL — capable of conducting analysis from start to finish.

• Strong foundation in statistics — including sampling, hypothesis testing, variance, and an understanding of noisy metrics.

• Analytical mindset — beginning with the business question rather than the tool.

• Over 3 years of experience in analyst or data scientist positions, with at least one role in a product context.

• Background in quality analytics for machine learning systems — such as ranking, recommendations, and classification.

• Practical experience in evaluating LLM applications (RAG, agents, tool usage, judges).

• Experience in developing LLM agents — side projects, toy builds, and personal experiments are all relevant.

• AI-assisted coding is our standard authoring environment.

• Proficient in AI coding or genuinely eager to become proficient quickly.

• We value the quality of your output and the clarity of your thought process.

🏝️ Benefits

• Make a significant impact on the product.

• Join our upward trajectory.

• Continuous opportunities for personal and professional growth.

• Empowered to make a meaningful impact on our evolving product.

• Flexibility to travel and work remotely or in a hybrid model throughout Europe.

• Become a holder of stock options.

• Stock Options Program available to all team members.

• Ongoing support and care to ensure your experience at Finom is both successful and fulfilling.

• Work & Swim Program in Cyprus to promote work-life balance.