Software Development Engineer in Test – Agentic Automation

This is a fully remote position, open to applicants in Canada.

📋 Description

• Create, develop, and sustain automated testing suites and frameworks for distributed, cloud-based systems.

• Enhance and expand agent-centric test generation and analysis tools integrated within major testing frameworks.

• Develop AI-driven workflows and agent skill files to automate manual testing tasks, including test data preparation, failure analysis, and release validation.

• Oversee evaluation suites, golden datasets, and scoring pipelines for AI agents, assessing correctness, tool utilization, trajectory, faithfulness, safety, latency, and cost.

• Implement CI evaluation gates to ensure that modifications to prompts, models, tools, or orchestration are assessed prior to release.

• Incorporate production failures back into evaluation datasets.

• Identify and address unreliable AI outputs, such as hallucinated assertions, flaky or redundant tests, incorrect failure analysis, and judge bias.

• Design, execute, and analyze performance, load, and stress tests to identify bottlenecks and validate scalability across distributed services.

• Validate the integration of platform systems, services, data, and applications, as well as customer-facing features comprehensively.

• Evaluate product, system, and agent designs for testability, evaluability, flaws, and enhancement opportunities.

• Share expertise and contribute to the team's quality and evaluation standards.

⛳️ Requirements

• Bachelor’s degree in Computer Science or equivalent experience (OOP, data structures, algorithms, distributed systems).

• Over 5 years in software quality, including a minimum of 3 years of hands-on experience as an SDET or in test automation development.

• Strong programming capabilities in at least one of Java, Node.js, or Python, with a preference for Python for evaluation and agent tasks.

• Demonstrated experience in designing, building, and maintaining test frameworks integrated with CI/CD pipelines (e.g., Jenkins, GitLab CI/CD).

• Experience in testing web services and APIs (REST, gRPC, JSON).

• Practical experience with performance and load testing using tools like Locust or k6, including the design of realistic workloads and interpreting results (latency percentiles, throughput, error rates).

• A skeptical, verification-first approach towards AI output, with discernment regarding when to apply deterministic assertions and when to utilize probabilistic evaluations.

• Proven experience in leveraging AI technologies to enhance decision-making, streamline workflows, and improve efficiency, driving business results.

• Must possess legal entitlement to work in Canada.

• Subject to background checks, including criminal record, credit, and/or reference checks.

• Bonus: Experience in end-to-end eval-driven development, including release gating, trajectory, tool-use evaluations, and calibrating LLM-as-judge against human review.

• Bonus: Experience in prompt engineering and context or tool design for agents.

• Bonus: Familiarity with integrating performance tests into CI/CD and utilizing observability tools (e.g., Grafana, Prometheus, CloudWatch).

• Bonus: Background in cybersecurity, threat intelligence, malware analysis, or sandboxing/detonation environments.

🏝️ Benefits

• Leading market compensation and equity awards.

• Comprehensive physical and mental wellness programs.

• Competitive vacation and holiday allowances for relaxation.

• Paid parental and adoption leave.

• Professional development opportunities available for all employees, regardless of their level or role.

• Employee Networks, local neighborhood groups, and volunteer opportunities to foster connections.

• Dynamic office culture equipped with world-class amenities.

• Variable/incentive-based compensation.

• Equity participation.

• Equal pay for equal work.