Director, AI Evaluation

This is a fully remote position, open to applicants in Pennsylvania.

📋 Description

• The Director of AI Evaluation is responsible for defining, demonstrating, and maintaining quality across the entire AI portfolio at Geisinger, encompassing both internally developed models and vendor-supplied systems.

• Oversees the team involved in building and validating AI systems: this includes data scientists who create production-level machine learning and refined AI systems, as well as senior analysts who assess their performance.

• Establishes quality standards, leads the team in enforcing these standards, and communicates findings to the VP of AI, executive leadership, and the board.

• Responsible for the methodology that ensures initiatives meet these standards, including pre-production validation and ongoing production monitoring.

• Attracts and retains top technical talent, fosters career development, and leads the evaluation team.

• Offers practical technical guidance to program teams while they design validation studies, conduct equity audits, develop monitoring plans, and create escalation playbooks.

• Manages the evaluation toolkit and reusable playbooks and templates, enabling each new program to progress more swiftly than its predecessor.

• Links each AI initiative to the outcomes it was intended to enhance, using a pre-launch baseline for reference.

⛳️ Requirements

• Experience in people leadership, including managing, developing, and cultivating technical staff; focuses on building cohesive teams rather than merely leading projects.

• Strong background in experimental design and causal inference, with the ability to discern which methods are appropriate for various scenarios.

• Practical experience in designing and executing model evaluation studies in real-world production environments.

• Familiarity with evaluating large language models (LLM) or generative AI systems, or equivalent experience with complex machine learning systems characterized by ambiguous or noisy ground truth.

• Demonstrated capability to convert unclear failure modes into specific, defensible evaluation frameworks and monitoring metrics.

• Proficient in Python and SQL, with a working knowledge of contemporary ML tools and cloud-native data environments.

• Experience in assessing fairness and equity within machine learning systems.

• Excellent written communication skills, as the role involves producing evaluation memos and specifications that are relied upon by non-technical decision-makers.

• Experience in healthcare, clinical settings, or regulated industries is highly preferred.

🏝️ Benefits

• We provide healthcare benefits for both full-time and part-time roles starting from day one, which includes vision, dental, and domestic partner coverage.

• We promote a collaborative, cooperative, and collegial work environment.