Senior Evaluation Algorithm Engineer – APAC

This is a fully remote position, open to applicants in Australia, +2 more countries.

📋 Description

• Create comprehensive evaluation plans for large language models (LLMs) applicable to dialogue, financial trading, and various business contexts.

• Develop systems and rubrics for evaluation metrics that yield quantifiable, reproducible, and interpretable outcomes.

• Oversee the design and construction of evaluation datasets, which includes defining evaluation dimensions, ensuring scenario coverage, establishing annotation guidelines, and implementing quality control measures.

• Examine the limitations and failure modes of models, providing actionable recommendations for enhancements.

• Partner with algorithm and product teams to facilitate model iterations.

• Streamline and scale evaluation workflows through automation.

• Create sustainable evaluation platforms and toolchains.

• Work in collaboration with algorithm, product, and data teams to transform objectives into evaluation standards and research and development directions.

⛳️ Requirements

• A Master's degree or higher in Computer Science, Artificial Intelligence, Mathematics, Statistics, or related disciplines.

• A robust understanding of algorithmic principles and LLM fundamentals, including training and fine-tuning methodologies.

• Practical experience in LLM evaluation within a major technology firm.

• Proficiency in evaluating commercial deployments, rather than solely relying on academic or offline benchmarking.

• Knowledge of the complete pipeline from preparing evaluation data and designing rubrics to evaluation-driven research and development.

• Familiarity with human evaluations, model-based automatic evaluations / LLM-as-a-judge, and metric calculation.

• Capability to define evaluation dimensions and draft clear, actionable, and distinctive rubrics.

• Systematic oversight of evaluation data representativeness, annotation consistency, and the reliability of results.

• Expertise in Python programming.

• Experience in automating evaluation workflows, constructing benchmarks, or developing evaluation platforms.

• Ability to independently manage data processing, write evaluation scripts, and analyze results.

• Strong business acumen and effective communication skills.

• Bonus: Experience in evaluating dialogue systems, AI agents, or financial/trading LLMs.

• Bonus: Experience in creating high-quality AI training/evaluation data or data annotation systems.

• Bonus: Familiarity with Reinforcement Learning from Human Feedback (RLHF), reward models, or tasks related to preference data.

🏝️ Benefits

• Contribute to shaping the future of finance.

• Collaborate with top-tier talent in a user-focused global organization with a flat hierarchy.

• Engage in challenging projects within a fast-paced and innovative environment while enjoying autonomy.

• Flourish in a results-oriented workplace that provides opportunities for career advancement and ongoing learning.

• Competitive salary along with company benefits.

• Remote-first work-from-home arrangement (subject to variation based on role and team requirements).