Senior QA Engineer

This is a fully remote position, open to applicants in Canada.

📋 Description

• Assess requirement specifications and technical design documents to provide prompt and insightful feedback.

• Develop detailed, thorough, and well-organized test plans and test cases while reviewing and updating existing test cases.

• Mentor and guide junior QA Analysts.

• Estimate, prioritize, plan, and manage testing activities effectively.

• Design, create, and execute automation scripts utilizing open-source tools, TypeScript, Playwright, or similar automation frameworks (such as Cypress, Selenium).

• Leverage AI coding assistants and agentic tools, like Claude Code, to enhance test authoring, maintain automation scripts, and manage defect triaging.

• Construct, customize, and sustain reusable AI skills and agentic workflows that optimize repeatable QA tasks (including test generation, regression checks, and defect documentation).

• Implement spec-driven development practices, such as the AI-Driven Development Lifecycle (AI-DLC), to convert requirements into structured specifications that facilitate AI-assisted test generation and validation.

• Design and execute evaluations ("evals") for AI-driven features to assess accuracy, relevance, hallucination rate, and other quality metrics against specified test datasets.

• Utilize non-deterministic testing methods for AI/LLM-based features, employing statistical thresholds, sampling, and repeated-run analysis instead of fixed pass/fail assertions.

• Set up and oversee AI observability and evaluation tools (e.g., Arize AX/Phoenix, LangSmith, or similar) to monitor model/agent performance, identify drift, and flag regressions in production.

• Identify, log, document, track, and resolve bugs.

• Conduct thorough regression testing upon bug resolution.

• Proactively share knowledge with other QA Team members through documentation and presentations, including AI-assisted testing methodologies and tools.

• Perform miscellaneous duties as necessary.

⛳️ Requirements

• Demonstrated experience in both manual and automated QA testing of web applications.

• Proficiency in TypeScript/JavaScript, with hands-on experience in Playwright or a similar automation framework (Cypress, Selenium, WebdriverIO).

• Practical knowledge of AI coding assistants and agentic tools (e.g., Claude Code, GitHub Copilot) for developing, maintaining, or reviewing test automation.

• Familiarity with spec-driven development practices (e.g., AI-DLC / AI-Driven Development Lifecycle) and the role of structured specifications in AI-assisted planning, execution, and validation.

• Capability to create and maintain custom AI skills, prompts, or agent workflows to enhance QA procedures.

• Understanding of non-deterministic testing, recognizing variability in AI/LLM outputs across runs and the need for different validation strategies compared to deterministic UI/API testing.

• Experience in designing evaluation ("eval") datasets and metrics (accuracy, relevance, hallucination, toxicity, latency) for AI-powered features.

• Knowledge of AI observability and evaluation-monitoring tools such as Arize (AX/Phoenix), LangSmith, or Braintrust for tracking model/agent quality in production.

• Strong foundation in QA methodologies, the SDLC, and test planning/estimation.

• Excellent analytical, written, and verbal communication skills, with the ability to mentor junior QA Analysts.

🏝️ Benefits

• Fully remote-first work with adaptable work arrangements 🏡

• Comprehensive Health and Wellness Benefits, including retirement savings programs, eligibility for two bonus plans, generous leave, and extensive medical and dental coverage based on your location 🧘

• New Hire Equipment Allowance and monthly Flex Allowance to ensure your success 📦

• Unlimited opportunities for career advancement and internal mobility 🌱

• Employee-driven DE&I initiatives 🫶