Staff AI Software Developer in Test

This is a fully remote position, open to applicants in Europe.

📋 Description

• Create and implement foundational QA strategies and evaluation frameworks for assessing AI agents, their workflows, and processes.

• Design and conduct experiments to evaluate AI-driven QA processes, which includes prompt strategies, automation workflows, and agent performance.

• Benchmark systems across various engineering streams to assess accuracy, reliability, cost, and consistency of outputs.

• Develop methods for generating synthetic datasets and structured test data to facilitate reliable evaluation.

• Identify shortcomings in current testing methodologies and devise AI-based solutions or agents to fill these gaps.

• Construct proof-of-concepts (POCs) to investigate new testing methodologies and validate concepts.

• Organize and document product knowledge, covering user journeys, APIs, and business rules, to aid AI-assisted testing systems.

• Assess the cost, ROI, and effectiveness of various QA methodologies and automation strategies.

• Work closely with the QA Lead and engineering teams to incorporate AI-assisted QA into development workflows.

• Ensure that AI-driven testing processes yield consistent, reliable, and high-quality results.

• Maintain an active role in developing tools, experiments, and testing infrastructure.

⛳️ Requirements

• Strong background in SDET and test automation.

• Experience in building and maintaining automated testing frameworks and QA tools.

• Hands-on experience in creating proof-of-concepts or experimental systems.

• Familiarity with evaluating or working alongside AI systems, LLMs, or agent-based workflows.

• Ability to design structured experiments and evaluation methodologies.

• In-depth knowledge of testing fundamentals, including UI and API automation tools like Cypress, Playwright, or similar.

• Experience in analyzing systems and pinpointing opportunities for process improvements through automation or AI.

• Ability to engage in hands-on work while also contributing to process design and strategic enhancements.

• Preferred Experience:

• Experience in AI-focused companies or startups.

• Practical experience in building or experimenting with AI agents or AI-driven automation workflows in Quality Assurance.

• Experience in designing evaluation frameworks for AI systems.

• Experience with synthetic data generation or test data modeling.

• Familiarity with knowledge graph systems or structured product knowledge mapping.

• Experience in evaluating cost efficiency and ROI of engineering or QA initiatives.

• Experience in dynamic startup environments.

🏝️ Benefits

• None specified in the job posting.