Senior AI Quality Engineer

This is a fully remote position, open to applicants in Canada.

📋 Description

• Develop and uphold evaluation frameworks for LLM outputs and agentic workflows, which encompass regression suites, golden datasets, and scoring rubrics.

• Create testing harnesses that identify hallucinations, tool-calling failures, prompt regressions, and unsafe or off-policy behaviors prior to production deployment.

• Establish quantifiable quality standards for agent reliability, including task completion, factual accuracy, latency, cost, and reasoning quality.

• Embed evaluation runs into CI/CD processes so that changes to models, prompts, and agents are managed similarly to code changes.

• Collaborate with engineers on observability and tracing for agent operations, ensuring that failures are understandable rather than enigmatic.

• Contribute to conventional APIs and end-to-end automation, integrating AI features within broader product workflows.

• Assist in developing the team's collective playbook for testing AI features, and mentor other QA engineers as agentic work proliferates across scrum teams.

• Engage actively as an individual contributor, reporting to the QA Manager.

• Collaborate with product, engineering, and the larger QA team on LLM-driven and agentic workflows.

⛳️ Requirements

• Over 6 years of experience in QA, SDET, or test automation, with proven success in delivering automation for production.

• Practical experience in testing LLM-based or agentic systems, including building evaluations, utilizing LLM-as-judge methodologies, prompt regression testing, or analyzing agent trajectories.

• Previous experience in a shift-left, embedded QA model.

• Proficiency with at least one modern automation stack (such as Playwright, Cypress, or similar) and a typed language, preferably TypeScript; Python is also acceptable.

• Extensive experience with test frameworks like vitest, jest, or pytest, and the ability to construct custom test harnesses rather than solely relying on pre-existing suites.

• An API-first testing approach, which includes familiarity with REST and Postman or similar tools.

• Expertise in HTTP-level API testing, including using recording proxies and analyzing service-to-service traffic.

• Understanding of CI/CD pipelines, with GitHub Actions being a plus, and knowledge of integrating evaluations into these workflows.

• Familiarity with cloud secret management and a disciplined approach to managing sensitive test data in restore-from-production environments.

• Capability to reason clearly about probabilistic systems, including variance, sample sizes, confidence intervals, and distinguishing between signal and noise in flaky results.

🏝️ Benefits

• Medical

• Dental

• Vision

• Flexible Spending Accounts

• PTO

• Paid and Floating Holidays

• 401k with Company match and immediate vesting

• Company-funded Life Insurance

• Employee Assistance Programs

• No-cost Mental Health Benefits

• Annual Bonus Opportunity of 10%