AI/ML Research Engineer, LLM Post-Training & Evaluation

This is a fully remote position, open to applicants in Canada.

📋 Description

• Create and develop the pipelines and tools that link data, evaluation, and post-training processes.

• Assist customers and internal teams in translating evaluation results into quantifiable model enhancements.

• Establish fine-tuning workflows such as supervised fine-tuning and preference-based optimization.

• Incorporate evaluation harnesses into model development cycles.

• Enhance the reliability and throughput of experiments.

• Aid in advanced evaluation scenarios, including long-context, cross-modal, and dynamic multi-turn interactions.

• Contribute to Innodata’s internal research and development initiatives, including benchmark datasets, evaluation frameworks, and reusable infrastructure for model evaluation and post-training experimentation.

• Design, build, and enhance LLM training and post-training pipelines, covering data ingestion, preprocessing, fine-tuning, evaluation, and experiment tracking.

• Implement and refine evaluation systems for LLMs and multimodal models.

• Integrate human-in-the-loop and AI-augmented evaluation signals into model development workflows.

• Develop robust infrastructure and tools for reproducible experimentation, metrics logging, and regression monitoring.

• Diagnose model behavior and pipeline issues.

• Collaborate with Language Data Scientists and Applied Research Scientists to convert evaluation frameworks into actionable systems.

• Work closely with customer technical stakeholders to grasp objectives, constraints, and success criteria; propose and execute technically sound solutions.

• Contribute to internal research and platform enhancement, encompassing benchmark frameworks, evaluation tools, and post-training workflow advancements.

• Advocate for best practices and standards in LLM training, evaluation, and quality assurance across various projects.

• Mentor junior engineers and participate in technical design reviews, documentation, and maintaining engineering rigor throughout the team.

⛳️ Requirements

• BS/MS/PhD in Computer Science, Machine Learning, AI, Applied Mathematics, or a related quantitative technical discipline (MS/PhD preferred).

• 2-3 years of pertinent industry or research engineering experience in ML/AI systems.

• Practical experience with LLM training, fine-tuning, and post-training, including at least one of the following:

• supervised fine-tuning (SFT)

• preference optimization (e.g., DPO or similar methods)

• RLHF/RLAIF-style workflows

• task- or domain-adaptation of foundational models.

• Strong programming capabilities in Python and experience in building production-quality ML code.

• Familiarity with modern ML frameworks (e.g., PyTorch, JAX, TensorFlow) and model libraries/tooling (e.g., Hugging Face ecosystem, vLLM, distributed training stacks).

• Experience in designing and implementing evaluation pipelines for LLM/ML systems, including metrics computation, dataset management, and experiment comparisons.

• Solid understanding of data pipelines and ML systems engineering, focusing on reproducibility, observability, and debugging.

• Experience with large-scale distributed ML systems and performance optimization for training/evaluation tasks (GPU/accelerator environments preferred).

• Experience with large-scale data processing and workflow orchestration to support model training and evaluation.

• Capability to collaborate directly with technical stakeholders, including research scientists, ML engineers, data engineers, and customer technical leads.

• Excellent written and verbal communication skills, with the ability to convey complex technical trade-offs to both technical and non-technical audiences.

🏝️ Benefits

• Competitive salary and performance-based incentives.

• Opportunities for professional growth and development.

• Flexible working hours and remote work options.

• Comprehensive health benefits including medical, dental, and vision coverage.

• Collaborative and innovative work environment.