Senior QA Automation Engineer

This is a fully remote position, open to applicants in Colombia.

📋 Description

• Integrate thoroughly within a Scrum team and engage in all ceremonies and associated artifacts.

• Take ownership of the comprehensive test strategy for the AI voice virtual agent encompassing LLM, speech, and telephony layers.

• Develop test scenarios for multi-turn dialogues, interruptions, ambiguous inputs, escalations, and failure recovery.

• Construct and uphold an evaluation framework for non-deterministic outputs utilizing golden datasets and semantic/rubric-based assertions.

• Conduct adversarial testing for hallucinations, incorrect grounding, guardrail bypasses, prompt injections, and unauthorized customer data disclosure.

• Set up and provision SIP devices, endpoints, and complete call scenarios.

• Diagnose failures across LLM, ASR, TTS, media path, and call routing layers.

• Enhance automated coverage and incorporate test suites into CI/CD release gates.

• Establish and monitor quality metrics such as task completion, containment, intent accuracy, transcription accuracy, latency, and escalation correctness.

• Keep documented test coverage in the TMS with traceability to business requirements.

• Communicate and escalate quality risks utilizing data and evidence.

⛳️ Requirements

• Over 5 years of experience in QA, with recent hands-on responsibility for a complex, integrated product.

• Practical experience in testing SIP/VoIP systems, including call flows, SIP signaling, media handling, softphones, or SIP trunking.

• Capability to troubleshoot using SIP traces or packet captures.

• Proven creativity in designing tests.

• Proficient in Python, capable of independently building and maintaining automation frameworks.

• Experience testing non-deterministic or probabilistic systems.

• Strong analytical abilities, including defining quality metrics and leveraging data for release decisions.

• Hands-on experience with formal TMS tools and Jira.

• Professional proficiency in English, both spoken and written.

• Reliable availability during US Eastern business hours.

• Advanced spoken English is required.

• Candidates must be located in LATAM.

🏝️ Benefits

• Opportunity for remote work.

• B2B employment with competitive gross compensation.

• Stable position with long-term growth prospects.

• Competitive salary reviewed annually based on performance.

• High-quality hardware provided.

• Continuous learning and career development opportunities.

• Compensation for professional training, seminars, and conferences.

• Referral program with incentives.

• Company-supported English classes.