AI/ML Evaluation Reviewer – Healthcare AI

This is a fully remote position, open to applicants in United States.

📋 Description

• Conduct independent assessments of technical evidence that supports healthcare AI applications, which encompasses model performance, validation methodologies, generalizability, reliability, proposed monitoring, and safeguards.

• Analyze model performance and validation techniques, including reference standards, sample sizes, statistical analyses, performance metrics, operating thresholds, and uncertainty reported.

• Evaluate the quality and representativeness of data, addressing issues such as data leakage, differences in subgroup performance, and generalizability across various patient populations and healthcare settings.

• Assess technical reliability, robustness, reproducibility, known failure modes, and model performance under atypical or out-of-distribution inputs.

• Review proposed evaluation plans, performance benchmarks, and success criteria related to pilot deployment.

• Evaluate technical safeguards, escalation processes, and human-in-the-loop mechanisms.

• Assess monitoring and change management strategies, including performance degradation detection, model behavior monitoring, updates, interventions, and rollback criteria.

• Collaborate with clinicians and other technical experts to address cross-disciplinary issues.

• Document strengths, limitations, gaps in evidence, and technical concerns using a standardized review framework.

• Offer clear, evidence-based recommendations and identify additional evidence or modifications necessary.

• Engage in reviewer calibration and quality assurance activities.

• Uphold independence and confidentiality while adhering to conflict-of-interest, confidentiality, and data-handling protocols.

⛳️ Requirements

• Minimum of 3 years of professional or applied research experience in machine learning, data science, biostatistics, clinical informatics, or a closely related field, with direct experience in evaluating or validating predictive models or AI systems.

• Proven understanding of model evaluation methodologies, including appropriate performance metrics, validation design, statistical uncertainty, threshold determination, and common sources of bias or methodological errors.

• Experience in assessing model performance on independent or held-out data and evaluating whether reported performance extends beyond the evaluation dataset.

• Hands-on experience with at least one relevant category of AI systems, such as clinical natural language processing, large language models, conversational AI, medical imaging, computer vision, or tabular predictive models.

• Ability to critically evaluate technical research, assessment reports, or validation studies, identifying methodological limitations or unsupported conclusions.

• Strong analytical writing skills, with the capacity to convey complex findings clearly to non-technical audiences.

• Capability to work independently while also contributing to multidisciplinary expert review and calibration efforts.

• Willingness to complete confidentiality agreements, conflict-of-interest disclosures, and training requirements for reviewers.

• Experience evaluating AI or ML systems utilized in healthcare, clinical decision support, digital health, or other high-stakes environments.

• Familiarity with clinical AI evaluation and reporting frameworks, including TRIPOD+AI, STARD-AI, CONSORT-AI, DECIDE-AI, or model cards.

• Experience assessing generative AI, LLM-based systems, or autonomous AI agents, covering aspects such as multi-turn behavior, reliability, hallucinations, and technical safeguards.

• Experience in robustness testing, adversarial evaluation, red teaming, or systematic failure-mode analysis.

• Knowledge of production monitoring, distribution shifts, model drift, version control, and change management practices.

• Experience in designing or evaluating human-in-the-loop systems or comparing model performance with human expert judgments.

• Familiarity with FDA guidelines for AI-enabled medical devices, clinical decision support, or software classified as a medical device.

• Experience in conducting independent technical reviews, scientific peer reviews, grant evaluations, or formal technology assessments.

• Experience collaborating with clinicians, researchers, or regulatory stakeholders on AI evaluation or deployment.

🏝️ Benefits

• Contract-based, per-review opportunity.

• Orientation for reviewers on the evaluation framework and project-specific requirements prior to the initial assignment.