Staff Quality Engineer

This is a fully remote position, open to applicants in Canada, +1 more country.

📋 Description

• Take ownership of the comprehensive quality strategy for the Automation and Intelligence Pillar pod's AI, including our Ivy product, overseeing everything from test architecture to release readiness.

• Create and implement evaluation frameworks for non-deterministic systems, which encompass LLM output quality, agent behavior, regression detection, and safety/guardrail testing.

• Construct dependable, scalable automated test suites and tools that cover functional, system, API, integration, UI, and regression testing, integrating these into the CI/CD pipeline.

• Set quality metrics and benchmarks for accuracy, latency, cost, and hallucination/failure rates, facilitating data-driven release decisions.

• Collaborate with Engineering Managers, Product Managers, Designers, ML/Data Science, and other engineering teams to synchronize requirements, priorities, and definitions of completion.

• Provide mentorship within the QE organization, advocate for QE best practices, drive process enhancements, and elevate AI testing across the company.

• Engage in technical planning, code reviews, and architectural decision-making, championing testability and reliability.

• Deliver timely, actionable feedback regarding test results to the pod, QA team, and stakeholders.

⛳️ Requirements

• Over 7 years of substantial, hands-on experience in software quality and test automation, demonstrating a history of establishing technical direction on complex systems.

• Extensive, hands-on experience with API & UI test automation frameworks such as Playwright, Cypress, Jest, or Junit.

• Proficiency in at least one programming language including TypeScript, JavaScript, Python, or Java, with a preference for TypeScript.

• Experience in testing AI/agentic systems (including evaluations, prompt/regression testing, and output validation) and utilizing AI to enhance automation (test generation, self-healing tests, and failure triage) with technologies like OpenAI and Anthropic.

• Familiarity with CI/CD tools such as Jenkins, GitHub Actions, or CircleCI.

• Strong understanding of scalable system design for web-based APIs and backend processing systems.

• Comprehensive knowledge of QA methodologies within a software company characterized by rapid release cycles.

• Demonstrated ability to establish test case design standards and coverage strategies, including edge/negative, compatibility, performance, and concurrency scenarios, and to manage a test case management practice (e.g., TestRail) that others can build upon.

• Understanding of React, particularly in relation to adding data-testids.

• Capability to investigate a codebase and employ technical tools to comprehend the technical implementation.

• Experience in coaching and influencing cross-functional teams regarding tooling, testing plans and strategies, and guiding them through execution.

🏝️ Benefits

• Highly competitive salary.

• Global retirement/401K/pension savings options.

• Generous paid time off packages, including days off for birthdays!

• Access to a comprehensive mental health care program.

• Health benefits provided from the first day of employment.

• Reimbursements for work-from-home expenses.

• Optional global WeWork membership for those seeking an alternative to their home office, along with hubs in London and Toronto.

• Comprehensive training and onboarding program.

• Support and coverage for personal development initiatives, including conferences, courses, and books.

• Access to StackAdapt programmatic courses and certifications for continuous learning.

• An excellent parental leave program.

• A friendly, welcoming, and supportive work culture.

• Exciting social and team events!