Applied AI Engineer

This is a fully remote position, open to applicants in Canada, +2 more countries.

📋 Description

• Develop and manage the runtime, tool invocation, and orchestration for production agent workflows.

• Create model selection and tiering strategies with cost and latency prioritized as key factors.

• Execute and optimize retrieval (RAG) and grounding techniques where they significantly enhance quality.

• Establish evaluation frameworks and golden sets to determine ship or no-ship decisions and identify regressions and drift.

• Design and implement assessment frameworks for evaluating model and agent quality.

• Construct guardrails and safety measures for retrieved content and tool outputs.

• Oversee production behavior, focusing on latency, cost, reliability, and failure modes.

• Collaborate with product teams to assess when an agent is an appropriate tool choice.

• Work alongside product teams and customers to validate results and outcomes.

• Demonstrate value with the smallest effective component, then expand on successful implementations.

• Monitor and measure impact to ensure evidence guides shipping decisions.

• Integrate automated quality assurance, including tests, quality gates, and regression checks, into the delivery pipeline.

• Establish evaluation and safety standards, participate in interviews, and mentor team members.

• Promote technical excellence, a focus on customer needs, and practical shipping practices.

• Collaborate with product teams across the portfolio and their respective roadmaps.

• Report directly to the Head of AI Platform Engineering.

• Contribute to team hiring and develop engineering patterns as an initial AI hire.

⛳️ Requirements

• Proven experience in delivering software powered by LLMs or ML to actual production users, beyond demonstrations and notebooks.

• Ability to evaluate effectiveness, not just focusing on prompting.

• Incorporate cost and latency considerations as design constraints from the outset.

• Communicate effectively with engineers, product collaborators, and customers.

• Emphasize outcomes and value, making pragmatic shipping decisions.

• Bonus: Experience with Agent or RAG systems in live production environments.

• Bonus: Expertise in model routing or demonstrable cost reduction.

• Bonus: Evaluation frameworks that inform decisions; LLM-as-judge calibrated against human judgment.

• Bonus: Experience in fine-tuning or distillation techniques.

• Bonus: Understanding of scenarios where using an agent may not be advisable.

• Bonus: Proficiency in Python or C#/.NET.

• Bonus: Experience in grounding AI within structured, well-modeled business data.

• Bonus: Skills in system and platform design, along with the ability to make trade-off judgments.

• Bonus: Background in building or operating enterprise-scale software, addressing reliability, security, and integration needs.

• Bonus: Effective use of AI coding assistants with discernment in overseeing outputs.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Flexible working hours and remote work options.

• Opportunities for professional development and continuous learning.

• Collaborative and innovative work environment.