Senior Business Intelligence Analyst – AI Evaluation, Monitoring

This is a fully remote position, open to applicants in Pennsylvania.

📋 Description

• The Senior BI Analyst acts as a collaborator with the business or clinical area being supported (AI Evaluation & Monitoring).

• Utilize analytical expertise, including knowledge of data structures, programming, and statistics combined with critical thinking.

• Gather, cleanse, and analyze data sets to address questions or resolve issues.

• Recognize and effectively communicate opportunities for improvement while aiding in the assessment of current AI/ML projects.

• Serve as the analytics specialist during meetings and discussions with stakeholders.

• Consult with AI stakeholders to collect evaluation and data requirements and define the scope of projects of varying sizes.

• Carry out evaluation tasks that ensure Geisinger's AI portfolio remains safe, efficient, and accountable.

• Work alongside internal partners and organizations to design and develop reporting solutions and analytical studies.

• Create, validate, and run complex queries and reports that address business inquiries for internal clients.

• Deliver analyses and reports to internal clients and stakeholders.

• Identify and assist in resolving data quality issues during data profiling, testing, or validation processes.

• Engage in strategic departmental initiatives and mentor junior BI developers.

⛳️ Requirements

• A minimum of 8 years of relevant experience* (Required).

• Preferably, experience in healthcare, clinical, or regulated-industry data.

• Proficiency in applied statistics, including hypothesis testing, confidence intervals, power and sample size, regression, and non-parametric methods.

• Experience with SQL in large, complex clinical data warehouses, including window functions, CTEs, debugging query plans, and performance optimization.

• Proficiency in Python for analysis, utilizing libraries such as pandas, numpy, scikit-learn, statsmodels, and other common evaluation libraries.

• Experience in production monitoring design and execution, including drift detection, performance decay tracking, and adoption and outcome metrics.

• Experience in executing evaluations of LLM and generative AI, including golden sets, judge-based scoring, and hallucination and grounding checks.

• Education: High School Diploma or Equivalent (GED) - (Required).

🏝️ Benefits

• Comprehensive healthcare benefits for full-time and part-time positions from day one, encompassing vision, dental, and domestic partners.

• Promotion of collaboration, cooperation, and collegiality.