Senior QA Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Take ownership of quality assurance for voice agents interacting with actual patients, providers, and caregivers.

• Develop testing and evaluation systems to support engineering advancements.

• Establish release criteria and identify necessary conditions for successful pre-launch and post-launch evaluations.

• Examine call recordings, transcripts, latency traces, and logs to pinpoint failures in speech-to-text, models, text-to-speech, telephony, and integrations.

• Generate reproducible defect tickets based on failed call scenarios.

• Create golden test sets, simulated calls, evaluated transcripts, classification and extraction accuracy tests, and structured-output validation mechanisms.

• Set tolerances and ensure reliable regression coverage for non-deterministic systems.

• Formulate test plans, regression suites, bug triage workflows, reproductions, and CI integration for the associated product.

• Oversee live calls post-release for any drift or degradation and escalate issues to engineers as necessary.

• Conduct red-team assessments on agents for prompt injection, off-script behavior, unauthorized identity claims, medical-advice risks, guardrails, and access controls.

• Evaluate products operated by Vi Operate, which integrates AI agents into the daily functions of health systems, providers, and pharmaceutical organizations; Augment supplies the foundational voice technology.

⛳️ Requirements

• Proven experience in testing a production voice agent that engages with users over the phone; skilled in diagnosing real failures and identifying the problematic layer.

• Familiarity with testing language-model systems in production and articulating performance evaluation methods.

• Strong background in classical QA processes, including test strategy formulation, test planning, regression suite development, bug triage, and release management.

• Experience in making non-deterministic output testable, along with the ability to articulate testing trade-offs.

• Proficiency in Python and test automation; capable of building tools and integrating them into CI pipelines.

• Sound judgment regarding defect severity and risks that may impact patients.

• Systems-oriented mindset with a focus on component interactions, unexpected inputs, and access-control vulnerabilities.

• Availability for remote work aligned with EST/CT time zones.

• Nice-to-have qualifications:

• Background in IVR, contact centers, or CCaaS quality, including familiarity with Genesys, NICE, Five9, Amazon Connect, Twilio, or similar platforms.

• Experience in audio and network condition testing, encompassing codecs, jitter, packet loss, barge-in, and interruption handling.

• Knowledge of LLM-as-judge frameworks, prompt regression testing, or simulated conversation evaluations.

• Experience in healthcare or pharmaceuticals with a comfort level working under HIPAA regulations.

• History of establishing a QA function.

• Experience in load and performance testing for real-time systems.

• Proficiency in Spanish sufficient to assess agent calls.

🏝️ Benefits

• Competitive salary and comprehensive benefits package.

• Opportunities for professional growth and development.

• Flexible work environment with remote options.

• Collaborative and innovative team culture.