
Staff AI Engineer, QA
Posted 12 hours ago

Posted 12 hours ago
This is a fully remote position, open to applicants in United States, +14 more locations.
• Develop the foundational AI system from start to finish, encompassing the service layer, data pipelines, and evaluation infrastructure, while taking ownership of its architecture.
• Design and implement evaluation infrastructure, which includes managing datasets, creating automated test harnesses, and enabling the replay of AI systems against historical workflows.
• Construct and manage evaluation datasets containing clinician-reviewed correct answers, ensuring they adapt as systems progress.
• Generate error metrics that are weighted by severity, focusing on errors that have clinical or financial implications.
• Establish production monitoring for accuracy, overrides, and safety concerns, providing actionable alerts for the engineering team.
• Develop adversarial tests for AI systems, incorporating hidden or malicious instructions in documents and fabricated content.
• Collaborate with product and clinical leaders to define accuracy thresholds and criteria for scaling AI capabilities or retracting them as necessary.
• Provide mentorship to a skilled team.
• Set quality standards for testing intelligent systems that influence patient care.
• A minimum of 7 years in software or machine learning engineering, with at least 3 years dedicated to evaluation, testing, or quality assurance for AI systems.
• Proven experience in designing and constructing complete systems from the ground up, successfully transitioning them from prototypes to production.
• A history of building and managing evaluation systems that are actively utilized in production environments, rather than solely for one-off analyses.
• At least 2 years of experience assessing large language model (LLM) systems in a production setting.
• Strong statistical expertise for measuring accuracy, including proficiency in sampling, confidence intervals, and error analysis.
• Over 1 year of experience in designing evaluation sets for document extraction, classification, or structured output.
• In-depth understanding of explainable AI (XAI) techniques and AI-powered tools (e.g., SHAP, LIME, Claude).
• Exceptional written communication skills, including the ability to produce design documents and explain intricate trade-offs to clinicians, engineers, and executives.
• A strong sense of curiosity and a readiness to adopt and utilize emerging AI tools actively.
• Comfortable with a variety of AI-enabled tools, including copilots, chat-based AI, and automation solutions.
• Experience in the healthcare sector or another highly regulated industry.
• Familiarity with AI-specific testing methodologies such as metamorphic testing, adversarial testing, or data validation with tools like Great Expectations.
• A track record of mentoring engineers and influencing technical direction across multiple teams.
• Numerous opportunities for career advancement.
• Comprehensive medical, dental, and vision coverage.
• 3 weeks of vacation alongside 5 personal days for rejuvenation.
• Employee stock ownership, RRSP program, and 401k matching.
• Opportunities to engage in community service.
• Flexible work arrangements tailored to your lifestyle.
• A relaxed work environment.
• Extensive benefits package.
• An award-winning company culture.
• Opportunities for continuous learning.
Timmons Group
Stone
Stefanini LATAM
Merit Medical Oncology
Get handpicked remote jobs straight to your inbox weekly.