Quality Assurance Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• Develop and sustain automated testing for AI voice and chat agents, encompassing everything from individual conversational exchanges to comprehensive roleplay scenarios.

• Assess the quality of LLM outputs - including accuracy, consistency, structured responses, language fidelity, and regression of prompts - utilizing evaluation frameworks (LLM-as-judge, golden datasets, and tolerance-based assertions for non-deterministic outputs).

• Verify RAG pipelines: ensuring relevance in retrieval, grounding/faithfulness, and overall answer quality.

• Evaluate voice pipelines: checking STT/TTS accuracy and real-time, low-latency performance across multiple languages.

• Implement automation across UI/E2E, API, and AI output layers, while constructing CI/CD processes from the ground up - including linting, type-checking, testing, evaluations, coverage gates, and deployment verifications.

• Take responsibility for release quality: develop regression strategies, detect issues early, and make clear go/no-go decisions.

• Conduct database and load/performance testing; enhance automation capabilities for both desktop and mobile applications.

• Assume control of and expand existing QA automation efforts, while reinforcing shared frameworks, tools, and QA processes/standards.

⛳️ Requirements

• Over 5 years of experience as a QA Automation Engineer, demonstrating a successful history of testing AI systems, beyond just traditional software.

• A builder mindset: have created testing frameworks, standards, and CI/CD-integrated automation from the ground up.

• Practical experience in testing conversational AI - either voice or chat agents - on actual, delivered projects.

• Strong understanding of LLMs, prompt engineering, and RAG - with the ability to design tests for non-deterministic outputs and assess retrieval/generation quality.

• Knowledge of voice pipelines (speech-to-text and text-to-speech) and methodologies for testing them through automation.

• Hands-on experience with LLM evaluation and observability tools, such as Langfuse, LangSmith, DeepEval, and RAGAS.

• Proficient in UI/E2E testing (e.g., using Playwright) and API test automation; experienced in constructing CI/CD pipelines from scratch.

• Comfortable with database testing, load/performance assessments, and automation for desktop and mobile testing environments.

• Strong discipline with Git, foundational cloud knowledge, and familiarity with the agile sprint process.

• Proven success in high-paced, fast-delivery settings - quickly adapts to unfamiliar systems and maintains a practical approach to process.

🏝️ Benefits

• Flexible work arrangements