Lead QA Engineer – AI Voice

This is a fully remote position, open to applicants in Poland.

📋 Description

• Develop the testing strategy and processes for voice AI, encompassing speech input through speech recognition (ASR), the LLM, tools and integrations, to speech output.

• Establish the approach for automation and its scalability, ensuring multilingual support for approximately 100 languages.

• Assess predictable behaviors such as intents, routing, and integrations, alongside non-deterministic AI behaviors.

• Analyze prompts and LLM responses using predefined test datasets and scoring criteria.

• Oversee UAT planning, including scenarios, test data, coordination with business testers, and recommendations for sign-off.

• Collect business acceptance criteria and translate them into quantifiable targets.

• Manage defects and risks, providing recommendations for release go/no-go decisions.

• Adjust plans and communicate options when there are development delays.

• Challenge stakeholders regarding readiness and quality assurance.

• Mentor the regular engineering team member.

⛳️ Requirements

• Minimum of 6 years of experience in software quality assurance.

• At least 2 years of experience in testing voice agents or conversational AI in a live delivery context.

• Proven experience leading end-to-end testing and UAT, functioning as the sole or lead QA with limited oversight.

• Background in working with international or distributed teams.

• Expertise in voice AI testing, covering aspects like speech recognition, speech output, latency, interruptions, accents, and background noise.

• Familiarity with voice quality metrics, including word error rate, intent accuracy, and task completion rates.

• Practical knowledge of the functioning and limitations of LLMs.

• Experience in test automation using Python or JavaScript/TypeScript, along with CI/CD and API testing tools like Postman.

• Proficient in utilizing cloud speech services such as Azure, Google, AWS, or equivalents to generate test calls.

• Hands-on experience with Jira or Azure DevOps, as well as a test management tool.

• Practical skills in troubleshooting through logs, transcripts, and call recordings.

• Ability to communicate in a structured, recommendation-focused manner with both technical and business stakeholders.

• English language proficiency at least at B2 level.

• Familiarity with voice testing platforms like Hamming, Cekura, Coval, or Botium is an advantage.

• Experience with LLM evaluation tools such as Promptfoo, DeepEval, or LangSmith is a plus.

• Knowledge of telephony fundamentals such as SIP, Twilio, or Azure Communication Services is beneficial.

• ISTQB certification is a plus.

• Experience in Responsible-AI testing, addressing PII, bias, and safety concerns is an advantage.

• Experience in validating customer journeys across various markets is a plus.

🏝️ Benefits

• Stable employment.

• “Office as an option” model: choose to work remotely or in the office.

• Workation in accordance with the workation policy.

• Great Place to Work® certified employer.

• Flexibility in working hours and preferred contract type.

• Comprehensive online onboarding program with a “Buddy” from day one.

• Collaboration with top-tier engineers and experts.

• Unlimited access to the Udemy learning platform from day one.

• Certificate training programs.

• Support for upskilling, capability development programs, Competency Centers, knowledge sharing sessions, community webinars, and over 110 training opportunities annually.

• Opportunities for internal promotion and career growth.

• A diverse, inclusive, and values-driven community.

• Autonomy in choosing your work style.

• Referral bonuses.

• Activities promoting well-being and health.

• Opportunities to contribute to charities and support environmental initiatives.

• Modern office equipment provided for you or available for borrowing, based on location.