Senior Applied AI/ML Scientist

This is a fully remote position, open to applicants in Canada.

📋 Description

• Collaborate with engineering teams to develop agents and their associated features.

• Establish standards for agent quality, evaluations, judges, guardrails, and necessary tools.

• Lead the selection, development, and validation of models.

• Design and implement A/B tests and other thorough experiments to assess customer impact.

• Set and refine quality standards for agents continuously.

• Work alongside engineers who are building and operating agents as well as evaluation infrastructure.

• Enhance agent-platform capabilities through prompt iteration, reusable judges, context-window management, tool utilization, and the adoption of cutting-edge models.

• Articulate AI capabilities, limitations, and quality to product, design, engineering, and leadership teams.

• Take ownership of evaluation and quality for agents, drive evidence-based model selection and improvement decisions, and contribute to AI quality roadmaps and methodologies.

⛳️ Requirements

• A minimum of 5 years of hands-on AI/ML experience with a proven record of evaluating, measuring, and enhancing deployed AI/ML or LLM systems with tangible impact.

• Experience in designing evaluation frameworks, constructing and calibrating judges or automated evaluators, and ensuring reliability for ML or LLM/agentic systems.

• Strong decision-making skills in model selection and validation.

• Expertise in rigorous experimentation and statistical analysis, including A/B testing and power analysis.

• Fundamental knowledge of ML and modeling, including proficiency in Python, common ML frameworks, transformers, and embeddings.

• Proficient in SQL and comfortable with large datasets.

• Ability to collaborate closely with product, design, and engineering teams and translate technical quality inquiries for stakeholders.

• Preferred: experience in evaluating production LLM-driven or agentic products.

• Preferred: experience in defining quality benchmarks or evaluation methodologies.

• Preferred: rapid prototyping experience with new AI/ML techniques.

• Preferred: familiarity with evaluation tools, observability (e.g., Datadog, Phoenix), and MLOps.

• Preferred: experience in influencing product roadmaps with insights regarding AI quality and feasibility.

• Preferred: understanding of model security, bias mitigation, and responsible AI practices.

• Must reside in Alberta, British Columbia, or Ontario, Canada.

🏝️ Benefits

• Comprehensive employer-paid health benefits, encompassing medical, dental, vision, life insurance, and disability coverage for the employee and eligible dependents.

• 25 vacation days per year.

• 5 paid sick days.

• All public holidays off.

• Company-wide Rest & Recharge days.

• Complimentary access to Modern Health, including coaching, therapy sessions, and digital wellness resources.

• Annual $950 CAD Lifestyle Spending Account for physical, mental, and financial wellness.

• One-time stipend of $550 USD-equivalent for home office setup.

• Monthly stipend of $50 USD-equivalent for internet expenses.

• No-cost, confidential access to financial experts through Your Money Line.

• Subsidized childcare and eldercare options available through Care.com.

• Company match for donations to eligible organizations.