Evaluations Team Lead

This is a fully remote position, open to applicants in Europe.

📋 Description

• Oversee the team that evaluates NEXUS, Fundamental's Large Tabular Model.

• Develop and manage a collaborative evaluation platform for engineering, research, and Applied AI initiatives.

• Continuously enrich and sustain models and curated datasets for independent benchmarking and exploration.

• Establish standards for metrics, data segmentation, leakage prevention, calibration, uncertainty, and benchmark contamination.

• Create reproducible pipelines with version-controlled inputs and artifacts.

• Incorporate regression checks into research workflows and release processes.

• Compare NEXUS with alternative approaches using fair tuning budgets, data accessibility, computational resources, and latency protocols.

• Assist Applied AI customers with proof-of-concept evaluations, providing tools, methodological guidance, and analysis.

• Assess predictive quality, latency, and costs across deployment configurations, task categories, and dataset characteristics.

• Translate findings into research priorities, release suggestions, and evidence-based improvement plans for customers.

• Recruit and mentor a small team, prioritize tasks, and maintain a hands-on approach to coding and experimental design.

⛳️ Requirements

• Proven experience managing evaluations for tabular ML systems utilized in production or critical customer decision-making.

• Strong statistical judgment in selecting metrics and validation schemes, estimating uncertainty, comparing models across datasets, and managing repeated experimentation.

• Practical experience in identifying leakage during preprocessing, feature engineering, joins, temporal dependencies, and related entity management across splits.

• Proficient in Python and SQL.

• Familiarity with scikit-learn and gradient-boosted trees.

• Experience in developing reliable ML tools or platforms used by other teams.

• Background in designing fair model comparisons, including hyperparameter optimization, resource limitations, and comprehensive latency assessments.

• Previous management experience, including hiring, technical mentorship, and performance evaluation, while remaining technically engaged.

• Effective written and verbal communication skills with researchers, engineers, and customer data scientists.

• Experience in evaluating tabular foundation models or AutoML systems (preferred).

• Knowledge of how model optimizations impact predictive quality and inference performance (preferred).

• Familiarity with relational or multi-table data, as well as Snowflake or Databricks environments (preferred).

• Experience in assessing automated or agent-driven ML workflows, including understanding issues that aggregated metrics may obscure (preferred).

🏝️ Benefits

• Competitive salary and equity compensation.

• Extensive health benefits for you and your dependents.

• Paid parental leave for all new parents, including those on adoptive and surrogate paths.

• Relocation assistance for employees transitioning to join the team at one of our office locations.

• A mission-driven, low-ego culture that promotes diverse perspectives, ownership, and a proactive approach.

• Fully remote position available in Europe or Israel.

• Monthly company gatherings in Barcelona, including 4 days of travel.