Senior Evaluation Algorithm Engineer

This is a fully remote position, open to applicants in Hong Kong, +2 more countries.

📋 Description

• Develop comprehensive LLM evaluation strategies for dialogue, financial trading, and various business contexts.

• Create evaluation metric frameworks and rubrics that yield measurable, reproducible, and comprehensible conclusions.

• Oversee the design and creation of evaluation datasets.

• Specify evaluation criteria and ensure scenario coverage.

• Set data annotation standards and maintain quality control procedures.

• Establish benchmarks that align with business requirements and possess discriminative capabilities.

• Examine model performance limits and identify failure modes.

• Offer actionable recommendations for improvement and partner with algorithm and product teams to facilitate model enhancements.

• Streamline and automate evaluation workflows.

• Develop sustainable evaluation platforms and toolchains.

• Collaborate with algorithm, product, and data teams to convert business and model goals into evaluation criteria.

• Transform evaluation insights into actionable R&D pathways and oversee their execution.

⛳️ Requirements

• A Master's degree or higher in Computer Science, Artificial Intelligence, Mathematics, Statistics, or relevant disciplines.

• A robust understanding of algorithmic principles and LLM methodologies, including training and fine-tuning processes.

• Practical LLM evaluation experience in a large technology organization.

• Engagement in commercial deployment evaluations rather than solely academic or offline benchmarks.

• Knowledge of the entire pipeline from evaluation data preparation and rubric formulation to evaluation-driven R&D.

• Experience with human evaluations, model-based automatic evaluations / LLM-as-a-judge, and metric calculations.

• Capability to delineate suitable evaluation criteria for various business scenarios.

• Proficiency in crafting clear, actionable, and effective rubrics.

• Systematic management of evaluation data representation, annotation consistency, and result dependability.

• Strong programming skills in Python.

• Background in automating evaluation workflows, constructing benchmarks, or developing evaluation platforms.

• Ability to manage data processing, evaluation script creation, and result analysis independently.

• Strong business acumen and effective communication skills.

• Aptitude for translating evaluation results into clear improvement strategies and fostering cross-team collaboration.

• Experience in evaluating dialogue systems, AI agents, or financial/trading LLMs is a plus.

• Experience in creating high-quality AI training/evaluation data or data annotation systems is advantageous.

• Familiarity with RLHF, reward models, or preference data-related tasks is a bonus.

🏝️ Benefits

• Competitive salary and comprehensive company benefits.

• Flexible work-from-home arrangements (may vary based on the business team's needs).

• Opportunities for career advancement and ongoing learning.

• Collaborate with top-tier talent in a user-focused global organization with a flat hierarchy.

• Engage in unique, fast-paced projects with a high degree of autonomy in an innovative setting.

• Results-oriented work environment.