Senior Data Scientist – AI Evaluation & Quality

This is a fully remote position, open to applicants in Lithuania.

📋 Description

• Become a part of the AI Team that is advancing Finom’s AI products and technologies.

• Take ownership of and enhance the offline evaluation suite for various products, which includes managing datasets, judges, and metrics.

• Develop and oversee online quality dashboards that track resolution rates, customer satisfaction (CSAT), thumbs up/down feedback, LLM-as-judge signals, error rates, and latency.

• Close the feedback loop in production by analyzing failure patterns from real traffic and converting them into regression cases.

• Suggest improvements to Product and domain experts.

• Strengthen evaluation methodologies, focusing on judge stability and handling non-determinism.

• Convert metrics into actionable decisions through weekly meetings and clear trade-offs.

• Collaborate effectively with AI engineers, Product teams, and domain experts throughout the organization.

• Utilize Databricks, DeepEval, and Claude Code in your work.

⛳️ Requirements

• Proficiency in Python and SQL, with the capability to conduct end-to-end analysis.

• A strong foundation in statistics, covering aspects like sampling, hypothesis testing, and variance.

• An analytical mindset that concentrates on business-related inquiries.

• A minimum of 3 years of experience in analyst or data scientist roles, including at least 1 year within a product context.

• Experience in quality analytics for machine learning systems (preferred).

• Practical experience in evaluating LLM applications such as retrieval-augmented generation (RAG), agents, tool usage, and judges (preferred).

• Experience in constructing LLM agents (preferred).

• Familiarity with AI-assisted coding as the standard authoring environment.

• Willingness to quickly become proficient in Claude Code for SQL, Python, analyses, dashboards, and internal scripts.

🏝️ Benefits

• Access to continuous resources and opportunities for personal and professional development.

• Flexibility to travel and work remotely or in a hybrid model throughout Europe.

• Stock Options Program available to all team members.

• Ongoing support and care from the organization.

• A modern, friendly, and environmentally conscious corporate culture.

• Work & Swim Program: Spend one month in a corporate apartment in Cyprus.