AI QA Engineer – AI Test Architect

This is a fully remote position, open to applicants in United States.

📋 Description

• Define, implement, and continually enhance testing standards across functional, non-functional, and AI-specific dimensions from requirements through to production.

• Design and execute performance, load, and stress test suites that are integrated into CI/CD pipelines, ensuring release quality gates are met.

• Establish and manage LLM/AI evaluation pipelines that assess accuracy, hallucination rates, relevance, faithfulness, and safety.

• Validate non-deterministic AI outputs, including prompt variability, model regression, guardrails, and multi-step agent task completion.

• Integrate QA into planning, design reviews, and sprint ceremonies.

• Champion quality across engineering, product, and AI teams; conduct blameless post-mortems and define quality metrics.

• Utilize AI-assisted tools, self-healing automation, and intelligent test prioritization.

• Define and monitor post-release quality signals, model drift indicators, and SLO thresholds.

⛳️ Requirements

• Strong understanding of deterministic testing, including test planning, test case design, functional, regression, and exploratory testing, as well as defect lifecycle management and quality metrics.

• Extensive experience in writing and maintaining automated test suites using Playwright, Cypress, or similar frameworks.

• Proficient in at least one modern programming language, particularly TypeScript or Python, for developing robust test libraries.

• Practical experience in performance, load, and stress testing utilizing k6 or JMeter, including CI/CD integration and threshold-based quality gates.

• Working knowledge of AI non-determinism and testing LLM-based features for hallucination, consistency, safety, and latency.

• Familiarity with LLM evaluation concepts, including BLEU, ROUGE, and LLM-as-judge patterns.

• Experience with at least one evaluation framework, such as DeepEval or RAGAS.

• Proven ability to integrate test suites into CI/CD pipelines such as GitHub Actions or CircleCI.

• Capability to take ownership of quality outcomes, set standards, raise risk flags, and influence cross-functional teams.

• Understanding of architectural patterns, microservices, and API testing using REST and gRPC with tools like Postman or custom frameworks.

• Experience with Docker and Kubernetes to create scalable testing environments.

• Nice-to-have: experience in testing agentic systems, AIOps/MLOps/LLMOps, accessibility or security testing, red teaming, adversarial input testing, prompt injection validation, and experience in startup or scale-up environments.

🏝️ Benefits

• Comprehensive benefits package.

• Competitive compensation structure.

• Flexible work arrangements available.

• A culture that promotes innovation, continuous learning, and significant growth.