Benchmark & Evaluation Engineer

This is a fully remote position, open to applicants in Philippines.

📋 Description

• Design and establish multi-system enterprise environments, which include warehouses, lakes, APIs, and databases, at scale.

• Construct environments across various domains such as healthcare, finance, and product analytics.

• Create calibrated, large-scale benchmark datasets that incorporate realistic noise, seasonality, drift, and synthetic PII.

• Evaluate query scalability and reasoning by utilizing benchmark datasets.

• Draft and validate benchmark tasks, golden SQL, and rubric-scored reasoning keys.

• Mitigate data leakage and ambiguity in benchmark tasks.

• Enhance grading systems through deterministic checks and LLM judges.

• Sustain regression-tracked leaderboards.

• Ensure the integrity of evaluation scoring.

⛳️ Requirements

• Expert-level proficiency in production-grade SQL, including CTEs, window functions, query plans, and cardinality.

• Deep knowledge of data modeling, including star/snowflake schemas, SCDs, and referential integrity.

• Strong skills in Python with experience in pandas, NumPy, and Polars.

• Practical experience in establishing and loading data into at least one major warehouse: Snowflake, BigQuery, Redshift, or Postgres.

• Demonstrated experience in designing evaluation sets or benchmark tasks.

• Familiarity with golden-answer/golden-SQL verification, rubric/LLM-judge calibration, preventing data leakage, and tracking regressions.

• Capability to create realistic multi-system configurations, including warehouses, data lakes, operational databases, and APIs.

• Experience handling large-scale, semi-structured, or dirty synthetic data.

• Domain knowledge in Healthcare/Health Insurance, Finance/FP&A, Product Analytics, or Supply Chain is a plus.

• Experience with LLM evaluation frameworks or streaming/data-lake stacks is a plus.

🏝️ Benefits

• Remote work opportunity in the Philippines.

• US shift schedule.