Lead Data Engineer

This is a fully remote position, open to applicants in Arizona, +8 more states.

📋 Description

• Take ownership of the ingestion, validation, editing, and storage pipelines for prescription drug event (PDE) records.

• Implement and uphold CMS-defined business rule validations, providing edit results back to plan sponsors.

• Design and maintain vendor reference data edits sourced from FDA, NCPDP, FDB, MediSpan, NPPES, and OIG.

• Deliver health plan and CMS status updates regarding PDE submissions.

• Construct, maintain, and enhance Apache Airflow and PySpark transformations utilizing S3, Redshift Serverless, Snowflake, and Databricks.

• Ensure compatibility with changes in the CMS data model.

• Monitor PDE adjustment trends and the rates of informational edits.

• Connect rejected PDEs with IDR records and provide beneficiary- and plan-level cost aggregations to PRS.

• Assist in CMS data transfer initiatives and new data-sharing integrations, including API and multi-cloud solutions.

• Maintain unit test coverage while supporting BDD and TDD methodologies.

• Track real-time processing metrics, including volumes, error rates, and reconciliation accuracy.

• Operate within the CMS Lean-Agile Release Train, delivering code through GitHub, Jenkins, JFrog Artifactory/XRay, SonarQube, and Snyk CI/CD pipelines.

⛳️ Requirements

• A Bachelor's degree in Computer Science, Data Engineering, or a related discipline.

• At least 8 years of experience in data engineering.

• A minimum of 4 years in building and operating high-volume batch ETL systems.

• Proficiency in Python and PySpark at an expert level.

• Production experience with Apache Airflow (AWS MWAA) at scale.

• Hands-on experience with AWS Redshift Serverless, Snowflake, and Amazon Athena.

• Practical experience with Databricks Notebooks and Jobs.

• Strong SQL skills for multi-source transformation, editing logic, and pipeline reconciliation.

• Familiarity with AWS S3, SQS, Lambda, and EventBridge.

• Experience with Linux environments, including RHEL, CentOS, and Amazon Linux 2.

• Proficient in Bash shell scripting.

• Experience managing pipelines under strict data quality, accuracy, and timeliness SLAs in a federal or healthcare setting.

• Knowledge of GitHub, Jenkins, and JFrog Artifactory for version control and CI/CD processes.

• Ability to pass CMS and internal background checks for public trust.

• Must reside in one of the designated states: Virginia, District of Columbia, Maryland, Tennessee, Florida, Arizona, Colorado, Oregon, or Texas.

• Employment is contingent upon contract award.

🏝️ Benefits

• 100% remote work.

• A supportive work/life balance.

• Opportunities for professional growth and development.

• Engaging in mission-driven healthcare IT work.

• A collaborative and supportive professional environment.