QA Engineer – GTM Applications

This is a fully remote position, open to applicants in India.

📋 Description

• Define and take ownership of the complete test strategy for agentic AI workstreams, setting quality standards that accommodate the probabilistic and non-deterministic characteristics of LLM-powered systems.

• Create and implement evaluation frameworks specifically for AI, focusing on hallucination detection, prompt quality scoring, agent task completion rates, and output fidelity against ground-truth references.

• Develop and maintain automated test suites in Python utilizing frameworks such as pytest and robot framework to cover unit, integration, and system-level scenarios across all components of the workstream.

• Establish RAG pipeline test coverage, including retrieval precision and recall, semantic relevance scoring, context fidelity, and end-to-end query-to-answer accuracy using tools like RAGAS.

• Design and construct QA automation tests using industry-standard tools and technologies, incorporating functional, regression, and end-to-end integration testing across interconnected systems and platforms.

• Create and execute Slack integration test suites to validate the accuracy of bot responses, Workflow Builder trigger fidelity, agentic Slack bot state management, and error handling under edge-case scenarios.

• Integrate automated tests into CI/CD pipelines (GitHub Actions, Copado, Jenkins) to ensure that every pull request meets a defined quality standard before merging.

• Design and perform performance and load tests for LLM-powered APIs, analyzing latency percentiles, token throughput, and degradation patterns under concurrent load conditions.

• Conduct security and adversarial testing, including prompt injection attempts, validation of output for sensitive data leakage, and collaboration with the DevSecOps team on SAST/DAST pipeline findings.

• Develop a regression strategy for non-deterministic outputs, establishing acceptable variance thresholds, snapshot-based comparisons, and statistical scoring methods that identify true regressions without false positives.

• Confirm the completeness of the observability stack by ensuring that distributed tracing, structured logging, SLOs, and AI-specific metrics (latency, token throughput, hallucination rates) are properly instrumented and alerting functions as intended.

• Collaborate with engineers and the AI Product Owner from requirements gathering through to sprint review, contributing to testability requirements, acceptance criteria, and definition-of-done checklists.

• Manage API contract testing for both internal and third-party integrations (Salesforce, Marketo, Snowflake, Gong, Clari, G-Suite) using tools like Postman or REST-assured.

• Oversee the defect lifecycle, including triage, severity classification, root cause analysis, regression prevention, and post-release quality retrospectives that inform the test strategy.

• Promote a shift-left quality culture by coaching engineers to write testable code, instrument their own unit tests, and view quality as a collective team responsibility rather than a final checkpoint at the end of the sprint.

⛳️ Requirements

• Bachelor’s degree in Computer Science, Engineering, Information Systems, or a related discipline.

• Over 6 years of QA or SDET experience, with a proven history of creating and maintaining automated test frameworks in production environments.

• Practical experience testing AI or ML systems, with a strong grasp of why non-deterministic outputs necessitate distinct evaluation strategies compared to traditional software.

• High proficiency in Python for test automation, including the design of reusable test utilities, fixtures, mocks, and data factories.

• Familiarity with test frameworks and tools such as pytest, Selenium, Playwright, REST-assured, Postman, or equivalent.

• Solid understanding of LLM behavior, including temperature effects, token limits, prompt sensitivity, and failure modes that affect test reproducibility.

• Direct experience in Salesforce QA testing, focusing on validation of Lightning Web Component (LWC) behaviors, Platform Event flows, and API integrations.

• Skilled in API testing across REST endpoints, including contract validation, schema compliance, payload verification, and error-path coverage.

• Experience with integrating automated tests into CI/CD pipelines to ensure quality gates are enforced automatically with every code modification.

• Knowledge of RAG evaluation metrics — faithfulness, answer relevance, context recall — and familiarity with tools such as RAGAS or LangSmith for structured AI output assessment.

• Experience with performance and load testing tools (Locust, k6, JMeter, or similar) to verify LLM-powered API behavior under realistic and peak loads.

• Basic understanding of security testing principles: prompt injection, output sanitization, awareness of OWASP top-10, and collaboration with DevSecOps tooling.

🏝️ Benefits

• Competitive compensation and equity awards.

• Comprehensive physical and mental wellness programs.

• Generous vacation and holiday policies for relaxation.

• Paid parental and adoption leave.

• Professional development opportunities available for all employees, regardless of their level or role.

• Employee Networks, local neighborhood groups, and volunteer opportunities to foster connections.

• Dynamic office culture featuring world-class amenities.

• Great Place to Work Certified™ globally.