Staff AI Engineer, QA

Posted 17 hours ago

This is a fully remote position, open to applicants in United States, +14 more locations.

📋 Description

• Develop the fundamental AI system from start to finish, overseeing everything from the service layer and data pipelines to the evaluation infrastructure and its architecture.

• Design and implement the evaluation infrastructure, which includes managing datasets, creating automated testing frameworks, and replaying AI systems against historical workflows.

• Construct and sustain evaluation datasets that include clinician-verified correct answers.

• Generate error metrics that are weighted by severity to prioritize mistakes that are clinically or financially significant.

• Create production monitoring systems for accuracy, overrides, and safety concerns, providing actionable alerts to the engineering team.

• Develop adversarial tests for AI systems, which may include concealed or harmful instructions within documents and fabricated content.

• Collaborate with product and clinical leaders to establish accuracy thresholds and criteria for scaling or retracting AI capabilities.

• Guide and mentor a talented team.

• Set quality standards for testing intelligent systems that affect patient care.


⛳️ Requirements

• A minimum of 7 years in software or machine learning engineering, with at least 3 years dedicated to evaluation, testing, or quality assurance for AI systems.

• Proven experience in designing and constructing complete systems from the ground up, transitioning them from prototypes to production.

• Demonstrated history of building and managing evaluation systems utilized in production environments, rather than just for one-off analyses.

• At least 2 years of experience assessing large language model (LLM) systems within a production context.

• Strong statistical expertise for accuracy measurement, including sampling, confidence intervals, and error analysis.

• Over 1 year of experience in designing evaluation sets for document extraction, classification, or structured output.

• In-depth understanding of explainable AI (XAI) techniques and AI tools (e.g., SHAP, LIME, Claude).

• Exceptional written communication skills, including the ability to produce design documents and convey complex trade-offs to clinicians, engineers, and executives.

• Experience in the healthcare sector or another highly regulated industry.

• Familiarity with AI-specific testing methodologies such as metamorphic testing, adversarial testing, or data validation using tools like Great Expectations.

• A history of mentoring engineers and influencing technical direction across various teams.

• A curious mindset and a readiness to adopt and utilize emerging AI tools actively.

• Comfortable with using AI-enabled tools such as copilots, chat-based AI, and automation solutions.


🏝️ Benefits

• Numerous opportunities for career advancement.

• Comprehensive medical, dental, and vision insurance.

• Three weeks of vacation along with five personal days to recharge.

• Employee stock ownership, RRSP program, and 401k matching.

• Opportunities to give back through community engagement.

• Flexible work arrangements to accommodate your lifestyle.

• Casual work environment.

• Extensive benefits package.

• Opportunities for learning and development.

• An award-winning workplace culture.

People also viewed

Timmons Group12 hours ago

Senior QA Analyst

QA flagQatar OnlyFull-timeQA Engineer (Quality Assurance)$80k – $120k/year
ApplyView job
Stone12 hours ago

Mid-Level Quality Assurance Engineer

BR flagBrazil OnlyFull-timeQA Engineer (Quality Assurance)
ApplyView job
Stefanini LATAM12 hours ago

QA Funcional

CL flagChile OnlyFull-timeQA Engineer (Quality Assurance)
ApplyView job
Merit Medical Oncology12 hours ago

Quality Engineer II

US flagUnited States OnlyFull-timeQA Engineer (Quality Assurance)
ApplyView job
Siemens Healthineers12 hours ago

Senior Commercial Product Quality Engineer – Post Market

US flagUnited States OnlyFull-timeQA Engineer (Quality Assurance)$121.3k – $166.7k/year
ApplyView job
Harris Computer13 hours ago

Staff AI Engineer, QA

US flagUnited States, +14 more locationsFull-timeQA Engineer (Quality Assurance)$120k – $160k/year
ApplyView job

Never miss a great job!

Get handpicked remote jobs straight to your inbox weekly.

Trusted by 7,400+ designers