Senior QA Automation Engineer, AI & LLM Testing

This is a fully remote position, open to applicants in Qatar.

📋 Description

• Design, develop, and uphold automated test suites for web applications, APIs, and AI-driven features.

• Assess LLM outputs for accuracy of responses, relevance, consistency, and potential hallucination risks.

• Create prompt tests and quality assessments utilizing Promptfoo or comparable LLM evaluation frameworks.

• Construct end-to-end UI automation using Playwright, adhering to page object models and principles of maintainable test architecture.

• Develop reusable Python scripts for API testing, automation tools, and LLM quality evaluation.

• Establish and manage Postman collections for REST API testing, which includes environments, assertions, and CI/CD integration.

• Leverage GitHub Copilot or similar AI coding assistants to enhance test coverage, expedite automation development, and pinpoint quality gaps.

• Work collaboratively with product managers, developers, and stakeholders to outline test strategies, convey risks, and enhance quality practices.

• Actively engage in Agile ceremonies, encompassing refinement sessions, sprint planning, retrospectives, and daily collaboration.

⛳️ Requirements

• Extensive experience in QA Automation and software testing.

• Practical experience with testing AI or LLM-driven features, including prompt testing, output validation, accuracy of responses, and mitigation of hallucinations.

• Familiarity with Promptfoo or other LLM evaluation tools/frameworks.

• Strong command of Playwright for end-to-end UI automation.

• Comprehensive understanding of page object patterns, reusable test components, and scalable test architectures.

• Proficient Python scripting abilities for automation, API helpers, and custom quality assessments.

• Experience with Postman and REST API testing, including collections, assertions, environments, and CI/CD integration.

• Hands-on experience utilizing GitHub Copilot or similar AI coding assistants.

• Experience in Agile settings such as Scrum or Kanban.

• Excellent communication skills, capable of articulating test strategies and quality risks to both technical and non-technical stakeholders.

• Nice to have: Experience in mobile test automation using Appium, XCUITest, or equivalent tools.

• Understanding of platform-specific testing considerations for iOS and Android.

• Experience with integrating automated tests into CI/CD pipelines.

• Familiarity with MCP or agent-assisted testing methodologies in Playwright.

🏝️ Benefits

• Employees have the option to work remotely.