Senior QA Engineer – AI Products, Customer Implementations

This is a fully remote position, open to applicants in Ukraine, +4 more countries.

📋 Description

• Take ownership of the testing strategy for one or more AI portfolio products, overseeing the process from requirements review to release sign-off.

• Design, develop, and sustain automated test suites for API, UI, integration, and regression testing, ensuring they are integrated into CI/CD pipelines.

• Evaluate multi-tenant platform functionality, including role-based access control (RBAC), tenant isolation, data access, and third-party integrations.

• Validate the behavior of AI agents, which encompasses conversation flows, workflow automation, scoring and coaching logic, as well as structured data extraction.

• Lead quality assurance for customer implementations by verifying configurations, connectors, knowledge bases, scoring rubrics, and workflow rules prior to launch.

• Design and implement User Acceptance Testing (UAT) plans in collaboration with client stakeholders and internal Delivery and Customer Success teams.

• Conduct exploratory and scenario-based testing based on actual customer workflows and observe early production behavior post-launch.

• Identify configuration errors, implementation defects, and product-level bugs, ensuring the relevant teams address them effectively.

• Manage the triage, documentation, and tracking of defects; collaborate with engineers on root-cause analysis and verification of fixes.

• Maintain comprehensive test plans, implementation checklists, and scalable QA playbooks.

• Transform recurring implementation challenges into automated regression tests and enhance QA processes.

• Report on release readiness, product quality, and implementation risks to Product, Engineering, Delivery, and client-facing teams.

⛳️ Requirements

• A minimum of 5 years of experience in software quality assurance, covering both product testing and customer-facing implementation or UAT activities.

• Proficient hands-on experience with test automation tools such as Playwright, Cypress, Selenium, or similar technologies.

• Strong background in API testing using tools like Postman, REST-assured, or equivalent resources.

• Excellent manual and exploratory testing abilities, with the skill to develop meaningful test scenarios from business and client requirements.

• Experience with testing multi-tenant and role-based access systems, focusing on data isolation and configuration-driven deployments.

• Ability to comprehend API and database schemas and collaborate with engineers on root-cause analysis.

• Familiarity with CI/CD pipelines and integrating automated tests into the development and release workflows.

• Comfortable engaging directly with external stakeholders during UAT and implementation validation.

• Strong grasp of software testing principles, defect lifecycle, regression strategies, and release readiness.

• Clear written and verbal communication skills, with the capacity to tailor communication for both technical and client-facing audiences.

• Preferred: experience testing AI/LLM-powered products, including accuracy evaluations, hallucination checks, prompt regressions, RAG quality, or AI output validation.

• Preferred: background in customer support, contact center, or business process outsourcing technology.

• Preferred: familiarity with AWS environments and testing cloud-based applications.

• Preferred: experience with observability, logging, tracing, or monitoring tools.

• Preferred: exposure to SOC 2, HIPAA, or PCI-DSS compliance environments.

• Preferred: experience testing integrations with Zendesk, Freshdesk, Intercom, telephony systems, or other customer support technologies.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance plans.

• Flexible work hours and remote work options.

• Opportunities for professional development and continuous learning.

• Supportive and innovative work environment.