Senior QA Automation Engineer – AI Products, Customer Implementations

This is a fully remote position, open to applicants in Ukraine, +1 more country.

📋 Description

• Take ownership of the testing strategy for one or more AI portfolio products, overseeing the process from requirements review to release sign-off.

• Design, develop, and uphold automated test suites for API, UI, integration, and regression testing, seamlessly integrated into CI/CD pipelines.

• Assess the behavior of a multi-tenant platform, focusing on RBAC, tenant isolation, data access, and third-party integrations.

• Confirm AI-agent functionality against specified requirements, which include conversation flows, workflow automation, scoring and coaching logic, and structured-data extraction.

• Lead quality assurance efforts for customer implementations by validating configurations, connectors, knowledge bases, scoring rubrics, and workflow rules prior to go-live.

• Create and implement User Acceptance Testing (UAT) plans in collaboration with client stakeholders as well as internal delivery and Customer Success teams.

• Conduct exploratory and scenario-based testing based on actual customer workflows.

• Monitor early production behavior post-launch.

• Differentiate between configuration issues, implementation defects, and product-level bugs, ensuring that the relevant teams address them appropriately.

• Triage, document, and track defects, collaborating with engineers on root-cause analysis and verification of fixes.

• Maintain scalable test plans, implementation checklists, and quality assurance playbooks.

• Transform recurring implementation challenges into automated regression coverage while enhancing QA processes.

• Communicate release readiness, product quality, and implementation risks to Product, Engineering, Delivery, and client-facing teams.

⛳️ Requirements

• Over 5 years of experience in software quality assurance, encompassing both product-level testing and customer-facing implementation or UAT activities.

• Extensive hands-on experience with test automation tools such as Playwright, Cypress, Selenium, or similar technologies.

• Strong background in API testing using tools like Postman, REST-assured, or other comparable alternatives.

• Proficient in manual and exploratory testing methodologies.

• Capable of developing meaningful test scenarios based on business and client requirements.

• Experience in testing multi-tenant and role-based access systems, including data isolation and configuration-driven deployments.

• Ability to read and comprehend API and database schemas and collaborate with engineers for root-cause analysis.

• Familiarity with CI/CD pipelines and integrating automated tests within development and release workflows.

• Comfortable engaging directly with external stakeholders during UAT and validation of implementations.

• Strong grasp of software testing principles, defect lifecycle, regression strategies, and release readiness.

• Excellent written and verbal communication skills, able to tailor communication for both technical and client-facing audiences.

• Nice to have: Experience in testing AI/LLM-powered products, including accuracy assessment, hallucination checks, prompt regression, RAG quality, or AI output validation.

• Nice to have: Background in customer support, contact center, or BPO technology.

• Nice to have: Familiarity with AWS environments and cloud-based application testing.

• Nice to have: Experience with observability, logging, tracing, or monitoring tools.

• Nice to have: Exposure to SOC 2, HIPAA, or PCI-DSS compliance environments.

• Nice to have: Experience testing integrations with technologies such as Zendesk, Freshdesk, Intercom, telephony systems, or other customer-support tools.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Flexible working hours and remote work opportunities.

• Professional development and continuous learning support.

• Collaborative and innovative work environment.