Quality Assurance Automation Engineer – AI

This is a fully remote position, open to applicants in Canada.

📋 Description

• Create, develop, and sustain robust automation frameworks specifically designed for testing AI functionalities, including outputs powered by large language models (LLMs), recommendation systems, and intelligent workflows.

• Formulate and implement testing strategies to validate AI model outputs, focusing on accuracy, relevance, consistency, and bias assessment across various input scenarios.

• Conduct functional, regression, performance, and scalability tests on AI-driven features, including API-level assessments of AI service integrations and comprehensive automated test suites.

• Work alongside AI engineers, data scientists, product managers, and UX designers to establish acceptance criteria and quality benchmarks for AI features throughout the development lifecycle.

• Identify, document, and monitor defects in AI behavior, including edge cases, hallucinations, and unexpected model responses, ensuring resolution through appropriate tools and workflows.

• Advocate for quality and responsible AI practices during the development lifecycle, contributing to prompt engineering reviews, model evaluation criteria, and AI safety guidelines.

• Develop and maintain CI/CD-integrated automation pipelines that consistently validate the quality of AI features, utilizing tools like Playwright, Claude, or similar frameworks.

• Engage in team stand-ups, sprint planning, retrospectives, and continuous improvement initiatives, fostering an AI-quality-first mindset in all discussions and helping to establish best practices for testing intelligent systems.

⛳️ Requirements

• 2-4 years of experience in QA automation engineering, with proven hands-on experience in building and maintaining automated testing frameworks.

• Proficiency in automation tools and frameworks (e.g., Playwright, Claude, or comparable) and API testing tools (e.g., Postman, REST-assured).

• Demonstrated experience in testing features or services integrated with AI/ML, with a strong understanding of the unique challenges posed by validating non-deterministic outputs, including LLM responses, probabilistic recommendations, and generative content.

• Bachelor's degree in Computer Science, Engineering, or a related technical discipline, or equivalent practical experience.

• Strong analytical and critical-thinking skills, with the capability to design test cases for complex, probabilistic AI behaviors.

• Excellent verbal and written communication skills, with the ability to effectively convey AI quality issues and their business implications to both technical and non-technical audiences.

🏝️ Benefits

• We’re a remote-first company with team members located across the USA, Canada, UK, and India!

• OnePlan has been honored as the Global Microsoft Partner of the Year in Project Portfolio Management in 2019, 2020, 2021, 2022, and 2023.

• We’ve been recognized as a "Strong Performer" in the latest Forrester Strategic Portfolio Management WAVE report.

• We provide comprehensive health, dental, and vision benefits, along with additional insurance options.

• Employer RRSP and 401K matching programs are available.

• Enjoy a fun, collaborative, and diverse work environment with regular health and team challenges to keep things light and enjoyable!