Staff AI Engineer, QA

Posted 12 hours ago

This is a fully remote position, open to applicants in United States, +14 more locations.

📋 Description

• Develop the foundational AI system from start to finish, encompassing the service layer, data pipelines, and evaluation infrastructure, while taking ownership of its architecture.

• Design and implement evaluation infrastructure, which includes managing datasets, creating automated test harnesses, and enabling the replay of AI systems against historical workflows.

• Construct and manage evaluation datasets containing clinician-reviewed correct answers, ensuring they adapt as systems progress.

• Generate error metrics that are weighted by severity, focusing on errors that have clinical or financial implications.

• Establish production monitoring for accuracy, overrides, and safety concerns, providing actionable alerts for the engineering team.

• Develop adversarial tests for AI systems, incorporating hidden or malicious instructions in documents and fabricated content.

• Collaborate with product and clinical leaders to define accuracy thresholds and criteria for scaling AI capabilities or retracting them as necessary.

• Provide mentorship to a skilled team.

• Set quality standards for testing intelligent systems that influence patient care.


⛳️ Requirements

• A minimum of 7 years in software or machine learning engineering, with at least 3 years dedicated to evaluation, testing, or quality assurance for AI systems.

• Proven experience in designing and constructing complete systems from the ground up, successfully transitioning them from prototypes to production.

• A history of building and managing evaluation systems that are actively utilized in production environments, rather than solely for one-off analyses.

• At least 2 years of experience assessing large language model (LLM) systems in a production setting.

• Strong statistical expertise for measuring accuracy, including proficiency in sampling, confidence intervals, and error analysis.

• Over 1 year of experience in designing evaluation sets for document extraction, classification, or structured output.

• In-depth understanding of explainable AI (XAI) techniques and AI-powered tools (e.g., SHAP, LIME, Claude).

• Exceptional written communication skills, including the ability to produce design documents and explain intricate trade-offs to clinicians, engineers, and executives.

• A strong sense of curiosity and a readiness to adopt and utilize emerging AI tools actively.

• Comfortable with a variety of AI-enabled tools, including copilots, chat-based AI, and automation solutions.

• Experience in the healthcare sector or another highly regulated industry.

• Familiarity with AI-specific testing methodologies such as metamorphic testing, adversarial testing, or data validation with tools like Great Expectations.

• A track record of mentoring engineers and influencing technical direction across multiple teams.


🏝️ Benefits

• Numerous opportunities for career advancement.

• Comprehensive medical, dental, and vision coverage.

• 3 weeks of vacation alongside 5 personal days for rejuvenation.

• Employee stock ownership, RRSP program, and 401k matching.

• Opportunities to engage in community service.

• Flexible work arrangements tailored to your lifestyle.

• A relaxed work environment.

• Extensive benefits package.

• An award-winning company culture.

• Opportunities for continuous learning.

People also viewed

Timmons Group11 hours ago

Senior QA Analyst

QA flagQatar OnlyFull-timeQA Engineer (Quality Assurance)$80k – $120k/year
ApplyView job
Stone11 hours ago

Mid-Level Quality Assurance Engineer

BR flagBrazil OnlyFull-timeQA Engineer (Quality Assurance)
ApplyView job
Stefanini LATAM12 hours ago

QA Funcional

CL flagChile OnlyFull-timeQA Engineer (Quality Assurance)
ApplyView job
Merit Medical Oncology12 hours ago

Quality Engineer II

US flagUnited States OnlyFull-timeQA Engineer (Quality Assurance)
ApplyView job
Siemens Healthineers12 hours ago

Senior Commercial Product Quality Engineer – Post Market

US flagUnited States OnlyFull-timeQA Engineer (Quality Assurance)$121.3k – $166.7k/year
ApplyView job
Bureau Veritas Group12 hours ago

QA/QC Engineer

US flagVirginia OnlyFull-timeQA Engineer (Quality Assurance)
ApplyView job

Never miss a great job!

Get handpicked remote jobs straight to your inbox weekly.

Trusted by 7,400+ designers