Senior AI Quality, Evaluation Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Take responsibility for independently demonstrating that retrieval functions correctly and that permissions are upheld.

• Develop the comprehensive question set with anticipated sources, passages, and acceptance criteria for relevance, grounding, and citation accuracy.

• Create negative entitlement tests to ensure restricted passages are not retrieved, triggering severity-one alerts upon failure.

• Establish freshness and retirement tests to monitor indexing delays and the elimination of outdated items.

• Automate the testing framework to operate on a schedule and following any content, configuration, or model adjustments.

• Publish results to the quality dashboard, complete with trend lines and regression alerts.

• Specify the QA gate that content analysts will apply to remediated materials.

• Execute the suite prior to pilot changes during weeks 7–9 and communicate missed targets along with root causes to the relevant owner.

• Compose the evaluation runbook and provide training for client stewards and platform owners.

⛳️ Requirements

• Over 3 years of experience in software quality engineering or test automation using Python, including the development of test harnesses that operate in CI on a regular basis.

• Practical experience evaluating a search or RAG system in a live environment, which includes creating ground truth sets, assessing relevance and grounding, and monitoring drift over time.

• Proven experience in designing security-oriented negative tests for authorization, entitlement, and data exposure.

• Proficient in working within AWS, utilizing Bedrock and OpenSearch APIs, interpreting CloudWatch data, and deploying scheduled jobs with infrastructure code.

• Strong written communication skills for reporting to client stewards and engineers.

• AWS Certified AI Practitioner or Machine Learning Engineer Associate preferred.

• Familiarity with RAG evaluation libraries and LLM-as-judge scoring with human calibration is preferred.

• Exposure to the contact center or knowledge management domains is preferred.

• Understanding of SOC 2 or ISO 27001 evidence practices is preferred.

• Proficiency in English is required.

🏝️ Benefits

• Influence real-world AI-driven projects across important industries, collaborating with clients ranging from innovative startups to large-scale enterprises.

• Join a global team that fosters collaboration across different continents and cultures.

• Experience an inclusive environment that emphasizes continuous learning, innovation, and ethical AI practices.