Senior Software Development Engineer in Test, QA - AI Systems

This is a fully remote position, open to applicants in Canada.

📋 Description

• Take charge of the quality assurance for Nexxa’s AI agent systems through both automated and manual testing.

• Design, construct, and uphold test frameworks, evaluation harnesses, and regression suites tailored for LLM-based agents.

• Develop automated evaluation and testing pipelines that are integrated into CI/CD quality gates.

• Create simulators, mocks, and sandbox environments for industrial systems and tool APIs.

• Implement API, contract, integration, and end-to-end tests for services, tools, and orchestration layers.

• Develop tests focusing on performance, load, reliability, failure injection, and resilience.

• Conduct exploratory testing of agent behavior in response to ambiguous requests, multi-turn sessions, and atypical operational scenarios.

• Review agent traces and outputs to assess reasoning quality, appropriateness of actions, and safety measures.

• Carry out manual regression and release verification in staging and customer-like environments.

• Conduct manual red-teaming assessments for prompt injection, jailbreaks, tool misuse, and unsafe actions.

• Assist in UAT and pilot rollouts alongside Forward Deployed Engineers.

• Compose test plans, test charters, bug reports, and manual testing documentation.

• Create and manage golden datasets and labeled test sets with version control and drift detection.

• Design evaluation rubrics and perform human-in-the-loop assessments.

• Define quality metrics such as groundedness, hallucination rates, task success rates, latency/cost trade-offs, and safety violations.

• Develop dashboards to monitor quality metrics over time.

• Test the full agent action loops, which include planning, tool selection, execution, recovery, and final output.

• Investigate and triage failures across models, prompts, tools/APIs, and orchestration logic.

• Establish quality standards, release criteria, and sign-off processes.

• Determine what aspects to automate and consistently transition repeatable manual processes into automation.

• Mentor engineers on testing strategies and test-driven practices specific to probabilistic LLM systems.

• Advocate for testability, observability, and tracing within agent architecture.

• Collaborate closely with ML engineers, backend engineers, and Forward Deployed Engineers.

⛳️ Requirements

• A minimum of 5 years in a role as an SDET, QA engineer, or software engineer with a strong emphasis on testing.

• Proven track record of owning test strategies and automation for complex distributed systems.

• Demonstrated experience in both manual and automated testing, including exploratory testing, test planning, and producing high-quality bug reports.

• Strong programming abilities, particularly in Python.

• Experience in building test frameworks and infrastructure from scratch.

• Practical experience with CI/CD systems such as GitHub Actions, GitLab CI, or Jenkins.

• Familiarity with containerized test environments, including Docker and Kubernetes.

• Experience in testing APIs and microservices, encompassing contract, integration, and end-to-end testing.

• Hands-on experience testing LLM-based products, chatbots, or AI agents.

• Understanding of testing generative systems and the limitations of deterministic assertions.

• Practical experience with evaluation frameworks or similar self-developed tools.

• Familiarity with LLM agents, including aspects like prompting, tool/function calling, context management, RAG, memory, and orchestration frameworks.

• Experience in designing test data and labeled datasets, including sourcing, sampling, and managing dataset drift.

• Understanding of LLM-specific failure modes, such as hallucination, prompt injection, context poisoning, tool misuse, goal drift, and non-determinism.

• Comfortable defining correctness in scenarios where no single right answer exists.

• Excellent written communication skills.

• Preferred: experience with human-in-the-loop evaluation workflows.

• Preferred: experience in performance/load testing and chaos/fault-injection testing.

• Preferred: background in ML/data science sufficient to analyze model evaluations and statistical significance.

• Preferred: experience in ML systems red-teaming or adversarial/security testing.

• Preferred: familiarity with observability and tracing tools for LLM applications and general observability stacks.

• Preferred: testing experience in industrial, IoT, or operational technology.

• Preferred: experience in UAT, pilot, or customer-facing validation.

• Preferred: experience in a startup or fast-paced team setting, establishing test and evaluation infrastructure from the ground up.

🏝️ Benefits

• Comprehensive salary and equity package.

• Significant opportunities for career growth and advancement.

• An innovative environment dedicated to transforming heavy industries through AI and automation.

• A collaborative culture that values innovation, discipline, and continuous improvement.