Software Development Test Engineer, AI & Automation

This is a fully remote position, open to applicants in California, +1 more state.

📋 Description

• Act as an integrated quality engineering partner within Keeper’s AI team.

• Collaborate with engineers and Product teams throughout the design and development phases.

• Establish test strategies, acceptance criteria, and automation requirements.

• Create and maintain automated testing frameworks for AI-driven applications, backend services, APIs, and related software systems.

• Develop both deterministic and probabilistic validation strategies for AI/ML outputs.

• Set evaluation criteria, tolerances, and quality thresholds for AI models and LLM-driven functionalities.

• Construct automated regression, integration, API, and end-to-end coverage in tandem with feature development.

• Develop test harnesses and evaluation workflows for prompts, models, agents, tool-calling workflows, and AI-integrated applications.

• Validate AI behavior across various models, prompts, inputs, and system configurations.

• Integrate automated validation and quality gates into CI/CD pipelines.

• Develop and maintain test datasets, mocks, fixtures, and synthetic data.

• Analyze model and application failures and produce clear, reproducible defect reports.

• Assess AI outputs for reliability, consistency, safety, and alignment with expected product behavior.

• Enhance application testability, observability, and diagnostic capabilities.

• Track test cases, execution results, and coverage using TestRail or similar platforms.

• Mitigate flaky tests while enhancing the speed, stability, and maintainability of automated test suites.

• Contribute to AI testing methodologies and best practices in automation.

• Utilize AI-assisted tools to enhance automation development, test generation, debugging, analysis, and documentation.

⛳️ Requirements

• Over 5 years of experience in SDET, software development, test automation, or another code-intensive quality engineering position.

• Practical experience testing AI/ML, LLM-powered, or model-integrated software systems.

• In-depth understanding of deterministic software testing and non-deterministic AI/ML validation.

• Experience in defining evaluation criteria, expected behaviors, tolerances, or scoring methodologies for AI-driven functionalities.

• Proficient programming skills in Python, Java, JavaScript/TypeScript, Rust, or another general-purpose programming language.

• Significant experience in designing, developing, and maintaining automated test frameworks.

• Experience in creating automated functional, integration, API, and end-to-end test coverage.

• Familiar with testing REST APIs and distributed application workflows using Postman, REST Assured, or similar platforms.

• Experience integrating automated tests into CI/CD pipelines using GitHub Actions, Jenkins, GitLab CI, or comparable technologies.

• Experience working in an Agile software development team with close collaboration among QA, Engineering, and Product.

• Strong grasp of shift-left testing, continuous quality, and testability principles.

• Familiarity with TestRail or similar test management platforms.

• Strong debugging and troubleshooting skills using logs, APIs, application telemetry, and development tools.

• Experience with Git-based development workflows, code reviews, and modern software engineering practices.

• Familiarity with AI-assisted development and testing tools such as Claude, ChatGPT, GitHub Copilot, or similar technologies.

• Strong analytical and problem-solving capabilities with the ability to evaluate variable AI behavior without relying solely on exact-match results.

• Excellent written and verbal communication skills, with the ability to collaborate closely with AI engineers, Software Engineers, Product Managers, and QA professionals.

• Bachelor’s degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience.

• Candidates must qualify as a U.S. Person, defined as a U.S. citizen or lawful permanent resident.

• Preferred: experience testing LLMs, agentic systems, retrieval-augmented generation, tool calling, or prompt-driven workflows.

• Preferred: experience in building automated model evaluation, benchmarking, or regression frameworks.

• Preferred: familiarity with AI evaluation metrics and techniques for assessing quality across non-deterministic outputs.

• Preferred: experience with Rust in production or test automation contexts.

• Preferred: experience in performance, reliability, security, or adversarial testing of AI-enabled systems.

• Preferred: experience with AI-assisted generation of automated tests or synthetic test data.

• Preferred: familiarity with Docker, Kubernetes, or cloud-based testing environments.

• Preferred: experience testing cybersecurity, identity, PAM, or other security-sensitive software.

• Preferred: experience in developing shared automation frameworks or internal testing tools utilized across engineering teams.

🏝️ Benefits

• Medical, Dental & Vision (including domestic partnerships).

• Employer Paid Life Insurance & Employee/Spouse/Child Supplemental life insurance.

• Voluntary Short/Long Term Disability Insurance.

• 401K (Roth/Traditional).

• A generous PTO plan that recognizes your commitment and seniority (including paid Bereavement/Jury Duty, etc).