Principal AI Engineer

This is a fully remote position, open to applicants in Pennsylvania.

📋 Description

• Take ownership of the comprehensive technical vision for MRO Prodigy's AI framework — a production system utilizing RAG, generative AI, and structured data reasoning to automate responses to Healthcare Registry questionnaires.

• Establish the AI roadmap for Prodigy, balancing immediate customer commitments with foundational capability investments necessary for scaling the product to enterprise-level maturity.

• Assess and make decisions regarding the build, buy, or integrate options for AI functionalities — including foundation model selection, embedding approaches, retrieval architectures, and orchestration frameworks — and take responsibility for the outcomes of those choices.

• Act as the technical authority for all AI design choices related to Prodigy; document architecture decision records, set standards, and ensure that the architecture is defensible, auditable, and extensible.

• Design and enhance Prodigy's multi-modal retrieval pipeline, integrating unstructured clinical document ingestion with structured EHR/FHIR data to deliver accurate, citation-supported answers to registry questionnaire items.

• Create and improve the answer generation layer — involving prompt engineering, context construction, grounding strategies, and output formatting — ensuring generated responses are clinically accurate and ready for audit.

• Manage the question routing and data source classification logic that aligns registry questions with the appropriate retrieval path, structured data field, or generation strategy.

• Develop and uphold the answer validation and confidence scoring framework, establishing the statistics and quality thresholds that determine when answers are automatically accepted or sent for human review.

• Remain actively involved and close to the work: lead direct ideation and feedback sessions with Prodigy's end users (abstractors, registry, and quality teams) as well as with production analytics and monitoring systems — transforming real usage signals into prioritized improvements that deliver tangible value, beyond just model metrics.

• Advance the feedback loop architecture that captures human corrections and integrates them into ongoing model improvement — ensuring Prodigy progressively enhances with every customer interaction.

• Define the evaluation framework for Prodigy: how accuracy is assessed, how regression is identified, and what indicators prompt retraining or prompt adjustments.

• Set guidelines for hallucination detection and factual grounding tailored to clinical registry use cases, where answer precision has direct downstream compliance ramifications.

• Design AI infrastructure across GCP (Vertex AI, BigQuery, Dataflow) and AWS (Bedrock), ensuring the pipeline is scalable, cost-effective, and operationally transparent.

• Collaborate with data engineering to maintain high-quality, well-governed clinical and FHIR data inputs; establish feature engineering and chunking strategies that enhance retrieval accuracy.

• Define MLOps standards for Prodigy: including model versioning, deployment checkpoints, rollback procedures, drift monitoring, and audit trail requirements in line with HIPAA compliance.

• Serve as the AI technical mentor for the Prodigy squad and neighboring engineering teams — guiding developers on RAG patterns, LLM integration, responsible AI methodologies, and clinical data management.

• Work closely with Security and Compliance to ensure Prodigy's AI layer adheres to HIPAA regulations, including PHI handling in prompts, data residency, and model audit logging.

• Promote AI literacy across the wider engineering organization, assisting teams in understanding when and how to safely apply AI within a regulated healthcare environment.

• Partner with Product Management to translate the complexities of registry workflows and customer feedback into technically sound enhancements to AI capabilities.

⛳️ Requirements

• Bachelor's degree in Computer Science, AI/ML, or a related field; Master's or PhD is preferred - or equivalent depth demonstrated through deployed AI systems.

• Solid foundation in ML/data science/statistics theory with the capability to read research, evaluate applicability, and implement effectively.

• Practical experience in LLM integration: prompt engineering, grounding, citation, hallucination mitigation, and output validation adhering to clinical accuracy standards.

• Familiarity with LangChain, LlamaIndex, or similar orchestration frameworks.

• Developed confidence scoring and auto-acceptance criteria that dictate when answers are directed to human review.

• Designed human-in-the-loop feedback systems that capture corrections and integrate them into model enhancement.

• Production experience with building RAG pipelines — including document ingestion, chunking, embedding model selection, vector store management, and retrieval evaluation.

• End-to-end ML lifecycle management in production: including versioning, deployment gates, drift monitoring, rollback procedures, and audit trails.

• Strong proficiency in Python and solid software engineering principles — including CI/CD, testing, and code review.

• Hands-on experience with vector databases (pgvector, Pinecone, Weaviate, or similar) and hybrid search techniques.

• Established evaluation frameworks that quantify accuracy, precision, recall, and F1 metrics to guide product decision-making.

• Extensive experience with AWS and GCP: including Bedrock, SageMaker, Vertex AI, BigQuery, and Dataflow.

• Familiarity with Azure is a plus.

• Experience in building pipelines that integrate mixed unstructured and structured data sources.

• Background in clinical NLP or healthcare AI — familiarity with medical terminology, document structures, and regulated accuracy standards is essential.

• Understanding of how HIPAA regulations apply to AI systems specifically: handling of PHI in prompts and embeddings, data residency, audit logging, and de-identification.

• Knowledge of AI governance practices: including bias detection, explainability, and responsible AI application in compliance-sensitive environments.

• Proven experience in owning an AI technical vision — not merely contributing to it.

• Ability to write an ADR, establish engineering standards, and ensure adherence across teams.

• Effective communication of trade-offs to both technical and non-technical stakeholders.

• A history of mentoring engineers and elevating AI maturity within a team.

🏝️ Benefits

• Eligible employees may also receive an annual cash bonus.

• Comprehensive benefits package, including medical, dental, vision, life insurance, and a 401(k) plan.