Applied Data Scientist, Health AI Evaluation, Datasets

This is a fully remote position, open to applicants in United States.

📋 Description

• Convert customer objectives — such as enhancing differential diagnosis, assessing a clinical note summarizer, evaluating a RAG-based medical literature assistant, or developing preference data for patient-centric chatbots — into detailed dataset specifications, taxonomies, rubrics, sampling plans, and acceptance criteria.

• Prioritize multimodal health AI: create training and evaluation datasets that encompass clinical text, medical images, waveforms, structured EHR data, claims, trial data, medical literature, patient communications, payer policies, drug information, and various clinical artifacts, along with use cases including clinical reasoning, medical QA, note summarization, medical coding, patient communication, utilization management, and literature synthesis.

• Develop evaluations for retrieval-augmented and source-grounded health AI systems, focusing on aspects such as evidence citation, faithfulness, handling of contraindications, adherence to guidelines, source freshness, and failure modes arising from incomplete, conflicting, or outdated context.

• Establish sampling strategies, labeling schemas, inter-annotator agreement targets, adjudication workflows, SME review patterns, and quality thresholds in collaboration with Language Data Scientists, clinicians, biomedical experts, and quality assurance teams.

• Create statistical and ML checks to ensure the reliability of healthcare datasets: implement stratified sampling across specialties and patient subgroups, conduct bias and representation analyses, detect leakage, perform distribution shift checks, estimate uncertainty, and analyze reliability metrics and subgroup performance.

• Collaborate with Applied Research Scientists and AI/ML Research Engineers to integrate datasets into evaluation and post-training workflows, including rubric-grounded LLM-as-judge prompts, regression suites, model comparison processes, experiment tracking, and feedback loops for model improvement.

• Assess health AI performance beyond mere accuracy: evaluate calibration, hallucination on safety-critical content, appropriateness of refusals, robustness in ambiguous situations, equity across patient subgroups, and safe transitions in agentic or integrated workflow systems. Consider concrete clinical workflow integration: understand where outputs fit into care delivery, what evidence a clinician or reviewer requires for trust, when uncertainty needs to be highlighted, and how patient-facing, clinician-facing, payer, pharma, and operational use cases differ in terms of risk.

• Manage data quality from intake to delivery, ensuring the integrity of de-identified clinical text, medical literature, synthetic cases, structured records, client policies, and knowledge bases, with a focus on PHI/PII handling, provenance, audit trails, version control, and compliance documentation.

• Keep abreast of developments in the health AI landscape — including regulatory updates like FDA guidance on AI/ML-enabled medical devices and EU AI Act health provisions, benchmark releases such as MedQA, MedMCQA, and HealthBench, along with emerging methodologies for clinical evaluation.

• Assist in customer discovery and proposal efforts by defining dataset programs, estimating annotation and SME review requirements, identifying regulatory or data access limitations, and clarifying methodology decisions to client clinical and ML leadership.

• Contribute to Innodata's internal intellectual property: develop reusable health-domain taxonomies, evaluation rubrics, golden datasets, clinical review playbooks, dataset quality checks, and methodology templates.

⛳️ Requirements

• A minimum of 5 years of data science experience, including at least 2 years in healthcare, clinical, biomedical, payer, provider, pharma, life sciences, or a similar regulated health data environment.

• Proficient understanding of healthcare data and standards: EHR structure, clinical documentation practices, ICD-10, CPT, SNOMED CT, LOINC, RxNorm, and at least a basic familiarity with FHIR, HL7, or analogous interoperability concepts.

• Practical experience in designing ML datasets, rather than merely consuming them: crafting annotation guidelines, estimating cohort sizes, establishing quality thresholds, creating QA checks, and delivering data that downstream teams can utilize for training or evaluation.

• Familiarity with LLM-based health AI workflows, including prompt design, rubric-based evaluation, retrieval-augmented generation, LLM-as-judge methodologies, model comparisons, and understanding the limitations of automated evaluations in clinical contexts.

• Strong skills in Python and SQL; proficiency with tools like pandas, scikit-learn, statsmodels or equivalent; and hands-on experience with modern LLM tools such as Hugging Face, evaluation frameworks, prompt development tools, or model APIs.

• Statistical knowledge encompassing sampling design, bias and fairness analysis, inter-annotator agreement metrics (such as Cohen or Fleiss kappa, Krippendorff alpha), confidence intervals, significance testing when applicable, error analysis, and the ability to challenge when data is being misinterpreted.

• Comprehensive understanding of healthcare privacy, compliance, and governance: HIPAA regulations, de-identification standards (Safe Harbor and Expert Determination), practical knowledge of handling PHI securely, auditability, access control, and documentation suitable for high-stakes or regulated AI initiatives.

• Capability to engage credibly with clinicians, biomedical SMEs, research scientists, engineers, technical solutions teams, annotators, and customer stakeholders.

• A preference for clinical realism: you would choose to develop a smaller dataset that accurately represents what clinicians, reviewers, patients, or care teams encounter rather than a larger dataset that appears impressive but fails in practical applications.

• A degree in a relevant discipline such as biostatistics, epidemiology, computational biology, health informatics, computer science with a health focus, statistics, or a clinical degree with quantitative training, or equivalent demonstrated experience.

• Clinical credentials are not mandatory, but candidates must be able to engage credibly with clinicians, biomedical SMEs, and health AI customers; candidates with MD, RN, PharmD, MPH, PhD, or health informatics backgrounds are particularly encouraged to apply.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Generous paid time off policy, including holidays and sick leave.

• Opportunities for professional development and continuous learning.

• Engaging work environment with a collaborative culture.