Senior QA Automation Engineer, AI Systems

This is a fully remote position, open to applicants in Qatar.

📋 Description

• Create and execute the validation framework to ensure RAG output quality, focusing on retrieval accuracy, citation correctness, and grounding.

• Develop automated testing suites for the AI Extraction Gateway, covering both Simple RAG and Complex RAG implementations.

• Collaborate with the BA and the Designated Subject Matter Expert to design and implement accuracy rubric test cases.

• Automate regression tests to assess confidence score calibration and the behavior of source weighting.

• Evaluate RBAC enforcement and access controls tailored to specific roles with filtered views.

• Ensure the completeness of audit logging and maintain document lifecycle traceability.

• Create and manage the test suite for the incremental ingestion pipeline.

• Assist in go/no-go decision processes by generating accuracy reports and compiling test evidence documentation.

⛳️ Requirements

• A minimum of 6 years in QA automation engineering, with a senior-level expertise required.

• Proficient test automation engineering capabilities.

• Python is preferred, with experience in pytest or a similar framework.

• Knowledge of API testing and contract testing methodologies.

• Adept at designing test frameworks for systems that are non-deterministic or probabilistic in nature.

• Background in agile/scrum environments, utilizing Jira for test management.

🏝️ Benefits

• Flexible work arrangements.