Applied Data Scientist, Health AI Evaluation, Datasets

This is a fully remote position, open to applicants in Canada.

📋 Description

• Transform customer objectives—such as enhancing differential diagnosis, assessing a clinical note summarizer, testing a RAG-based medical literature assistant, or generating preference data for patient-facing chatbots—into detailed dataset specifications, taxonomies, rubrics, sampling plans, and acceptance criteria.

• Center multimodal health AI as a primary objective: devise training and evaluation datasets encompassing clinical text, medical images, waveforms, structured EHR data, claims, trial data, medical literature, patient communications, payer policies, drug information, and other clinical artifacts, along with use cases like clinical reasoning, medical QA, note summarization, medical coding, patient communication, utilization management, and literature synthesis.

• Create evaluation frameworks for retrieval-augmented and source-grounded health AI systems, focusing on aspects such as evidence citation, faithfulness, handling of contraindications, adherence to guidelines, source freshness, and identifying failure modes arising from incomplete, conflicting, or outdated context.

• Establish sampling strategies, labeling schemas, inter-annotator agreement targets, adjudication workflows, SME review patterns, and quality thresholds in collaboration with Language Data Scientists, clinicians, biomedical experts, and quality assurance teams.

• Develop statistical and ML verification processes to ensure the reliability of healthcare datasets: implement stratified sampling across specialties and patient subgroups, conduct bias and representation analyses, detect data leakage, perform distribution shift checks, estimate uncertainty, and analyze reliability metrics and subgroup performance.

• Collaborate with Applied Research Scientists and AI/ML Research Engineers to integrate datasets into evaluation and post-training workflows, including rubric-based LLM-as-judge prompts, regression suites, model comparison processes, experiment tracking, and feedback loops for model enhancement.

• Assess health AI performance beyond mere accuracy: evaluate calibration, hallucination on safety-critical content, appropriateness of refusals, robustness in ambiguous situations, equity among patient subgroups, and safe transitions in agentic or workflow-integrated systems. Analyze how outputs fit into clinical workflows: where they fit in care delivery, what evidence clinicians or reviewers require for trust, when uncertainty should be revealed, and how patient-facing, clinician-facing, payer, pharma, and operational use cases differ in associated risks.

• Take ownership of data quality from initial source intake to final delivery, encompassing de-identified clinical text, medical literature, synthetic cases, structured records, client policies, and knowledge bases, with a strong focus on PHI/PII management, provenance, audit trails, versioning, and compliance documentation.

• Remain informed about the evolving health AI landscape—including regulatory changes such as FDA guidance on AI/ML-enabled medical devices and EU AI Act health provisions, benchmark releases like MedQA, MedMCQA, and HealthBench, as well as new clinical evaluation methodologies.

• Assist in customer discovery and proposal efforts by outlining dataset programs, estimating annotation and SME review workload, identifying regulatory or data-access limitations, and articulating methodology choices to client clinical and ML leadership.

• Contribute to Innodata's internal intellectual property: develop reusable health-domain taxonomies, evaluation rubrics, golden datasets, clinical review playbooks, dataset quality checks, and methodology templates.

⛳️ Requirements

• Minimum of 5 years of experience in data science, including at least 2 years in healthcare, clinical, biomedical, payer, provider, pharma, life sciences, or similar regulated health data environments.

• Proficient understanding of healthcare data and standards: familiarity with EHR structure, clinical documentation conventions, ICD-10, CPT, SNOMED CT, LOINC, RxNorm, and at least a working knowledge of FHIR, HL7, or comparable interoperability concepts.

• Practical experience in designing ML datasets rather than merely consuming them: crafting annotation guidelines, estimating cohort sizes, establishing quality thresholds, devising QA checks, and delivering data suitable for downstream training or evaluation.

• Knowledge of LLM-based health AI workflows, including prompt design, rubric-based evaluation, retrieval-augmented generation, LLM-as-judge methodologies, model comparison, and the limitations of automated evaluation in clinical settings.

• Strong proficiency in Python and SQL; adept with tools like pandas, scikit-learn, statsmodels or their equivalents; and familiarity with modern LLM tools such as Hugging Face, evaluation frameworks, prompt development tools, or model APIs.

• Statistical proficiency in sampling design, bias and fairness analysis, inter-annotator agreement metrics (Cohen or Fleiss kappa, Krippendorff alpha), confidence intervals, significance testing where relevant, error analysis, and the ability to challenge over-interpretation of data.

• Solid understanding of healthcare privacy, compliance, and governance: knowledge of HIPAA, de-identification standards (Safe Harbor and Expert Determination), practical handling of PHI safely, auditability, access control, and documentation suitable for high-stakes or regulated AI programs.

• Ability to engage effectively with clinicians, biomedical SMEs, research scientists, engineers, technical solutions teams, annotators, and customer stakeholders.

• A preference for clinical realism: you prioritize creating a smaller dataset that accurately reflects what clinicians, reviewers, patients, or care teams encounter over a larger dataset that appears impressive but is ineffective in practice.

• A degree in a relevant field such as biostatistics, epidemiology, computational biology, health informatics, computer science with a health focus, statistics, or a clinical degree with quantitative training, or equivalent demonstrated experience.

• Clinical credentials are not mandatory, but candidates must be able to work credibly with clinicians, biomedical SMEs, and health AI clients; candidates with MD, RN, PharmD, MPH, PhD, or health informatics backgrounds are particularly encouraged to apply.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Opportunities for professional development and growth.

• Flexible work environment with remote work options.

• Supportive and collaborative team culture.