
Staff AI Engineer, QA
Posted 17 hours ago

Posted 17 hours ago
This is a fully remote position, open to applicants in United States, +14 more locations.
• Develop the fundamental AI system from start to finish, overseeing everything from the service layer and data pipelines to the evaluation infrastructure and its architecture.
• Design and implement the evaluation infrastructure, which includes managing datasets, creating automated testing frameworks, and replaying AI systems against historical workflows.
• Construct and sustain evaluation datasets that include clinician-verified correct answers.
• Generate error metrics that are weighted by severity to prioritize mistakes that are clinically or financially significant.
• Create production monitoring systems for accuracy, overrides, and safety concerns, providing actionable alerts to the engineering team.
• Develop adversarial tests for AI systems, which may include concealed or harmful instructions within documents and fabricated content.
• Collaborate with product and clinical leaders to establish accuracy thresholds and criteria for scaling or retracting AI capabilities.
• Guide and mentor a talented team.
• Set quality standards for testing intelligent systems that affect patient care.
• A minimum of 7 years in software or machine learning engineering, with at least 3 years dedicated to evaluation, testing, or quality assurance for AI systems.
• Proven experience in designing and constructing complete systems from the ground up, transitioning them from prototypes to production.
• Demonstrated history of building and managing evaluation systems utilized in production environments, rather than just for one-off analyses.
• At least 2 years of experience assessing large language model (LLM) systems within a production context.
• Strong statistical expertise for accuracy measurement, including sampling, confidence intervals, and error analysis.
• Over 1 year of experience in designing evaluation sets for document extraction, classification, or structured output.
• In-depth understanding of explainable AI (XAI) techniques and AI tools (e.g., SHAP, LIME, Claude).
• Exceptional written communication skills, including the ability to produce design documents and convey complex trade-offs to clinicians, engineers, and executives.
• Experience in the healthcare sector or another highly regulated industry.
• Familiarity with AI-specific testing methodologies such as metamorphic testing, adversarial testing, or data validation using tools like Great Expectations.
• A history of mentoring engineers and influencing technical direction across various teams.
• A curious mindset and a readiness to adopt and utilize emerging AI tools actively.
• Comfortable with using AI-enabled tools such as copilots, chat-based AI, and automation solutions.
• Numerous opportunities for career advancement.
• Comprehensive medical, dental, and vision insurance.
• Three weeks of vacation along with five personal days to recharge.
• Employee stock ownership, RRSP program, and 401k matching.
• Opportunities to give back through community engagement.
• Flexible work arrangements to accommodate your lifestyle.
• Casual work environment.
• Extensive benefits package.
• Opportunities for learning and development.
• An award-winning workplace culture.
Timmons Group
Stone
Stefanini LATAM
Merit Medical Oncology
Get handpicked remote jobs straight to your inbox weekly.