Software Engineer in Test – AI Services, Healthcare

This is a fully remote position, open to applicants in Europe.

📋 Description

• Develop and establish model validation frameworks focusing on accuracy, precision, recall, and F1 scores across various clinical subgroups such as age, hearing loss severity, and device type.

• Create regression tests for model updates to identify silent accuracy degradation prior to any production deployment.

• Construct adversarial test cases that include edge cases, out-of-distribution inputs, boundary conditions, and clinically implausible inputs.

• Design and sustain API test suites for AI feature endpoints utilized by the PMS frontend and microservices.

• Develop contract tests between AI services and their consumers to prevent integration issues when models are updated or retrained.

• Assess latency, throughput, and the system's ability to gracefully degrade under load conditions.

• Validate error handling mechanisms related to model confidence thresholds and fallback behaviors when models are unavailable or provide low-confidence outputs.

• Collaborate with the AQA team on shared Playwright end-to-end coverage for AI-integrated UI flows.

• Create testing evidence for ISO 42001 AIM, ensuring traceability from AI system requirements through to test cases and results, supporting audit and certification processes.

• Design and implement model monitoring pipelines to track accuracy, confidence distribution, and prediction drift in production compared to baseline metrics.

• Participate in post-incident reviews when AI features impact clinical workflows unexpectedly or trigger regulatory flags.

• Incorporate insights from production monitoring back into the regression test suite to mitigate recurrence and enhance model robustness.

⛳️ Requirements

• Over 5 years of experience in software testing.

• At least 1 year of experience specifically testing AI/ML systems, data pipelines, or model-serving APIs.

• Proficiency in automation frameworks such as .NET and TypeScript / Playwright for AI-integrated UI coverage.

• Experience in API testing, including contract testing, latency profiling, and coverage of error scenarios.

• In-depth understanding of QA methodologies, processes, and CI/CD practices.

• Familiarity with service-oriented and microservice architectures.

• Knowledge of machine learning fundamentals, including train/test splits and evaluation metrics (precision, recall, F1, AUC).

• Strong problem-solving and debugging abilities, with experience in investigating both backend and frontend issues.

• Capacity to think from an end-user perspective and test accordingly, particularly when product requirements are minimal or evolving.

• Excellent communication skills in English (B2+ level).

🏝️ Benefits

• Opportunity for long-term engagement within a stable, growing SaaS company.

• A remote-first and asynchronous-friendly work environment with a flat organizational structure.

• A commitment to high standards for engineering and product quality.

• A team that prioritizes depth over hype, emphasizing production quality over flashy prototypes.

• A culture that values personal growth alongside business outcomes.

• A global team with members from Denmark, Ukraine, the UK, Poland, Canada, Australia, the USA, and beyond.