Machine Learning Engineer, Core Evaluations

This is a fully remote position, open to applicants in United States.

📋 Description

• Developing pipelines for evaluating models both in development and production stages.

• Creating user studies aimed at conducting subjective evaluations of models.

• Translating requirements into quantifiable metrics.

• Designing and implementing an automated evaluation dashboard to monitor model performance and compare results.

• Training new models to incorporate diverse evaluation metrics.

• Collaborating with the model team to enhance model design based on evaluation findings.

• Working with the data team to determine the types of data required to boost model performance.

• Coordinating with the product manager to ensure accurate measurement of product requirements.

• Contributing to the growth of the evaluation team as a founding member.

• Potentially leading the evaluation team in the future.

⛳️ Requirements

• Extensive experience and intuition in designing metrics that effectively capture model performance.

• Significant experience in designing user studies via Mechanical Turk or similar platforms.

• Strong background in model training and fine-tuning for evaluation purposes.

• Solid statistical knowledge and experience to compare evaluation results statistically and make informed decisions.

• Exceptional engineering and programming capabilities.

• Experience in training ASR and TTS models.

• Background in machine learning teams dealing with large-scale machine learning challenges (over 3 billion models with more than 1 million hours of data).

🏝️ Benefits

•