Senior Data Platform Engineer, Databricks, Python

This is a fully remote position, open to applicants in Arizona, +5 more states.

📋 Description

• Develop and maintain shared Python libraries along with configuration-driven frameworks to facilitate domain-team pipeline onboarding.

• Manage GitHub Actions pipelines and oversee the release process for promoting Databricks Asset Bundles from sandbox to production environments.

• Expand Terraform coverage for Azure and Databricks infrastructure.

• Provide support for the production platform and improve observability.

• Address incidents and conduct root cause analysis.

• Transform lessons learned into runbooks, automation solutions, and permanent fixes.

• Monitor performance, reliability, and delivery metrics from both data and platform viewpoints.

• Ensure platform and data controls are maintained using Unity Catalog.

• Assist in protecting sensitive data and ensuring governance within a regulated healthcare setting.

• Collaborate with domain teams to onboard data sources and address escalated issues.

• Convert recurring domain-team requirements into self-service features.

⛳️ Requirements

• A minimum of 3 years of progressive experience in software, platform, DevOps/SRE, or data engineering.

• At least 2 years of experience in building or operating cloud data platforms or production data systems.

• Proficient in Python, including capabilities in packaging, creating modular and testable code, and utilizing pytest.

• Experience with Git-based code reviews.

• Demonstrated ability in building CI/CD pipelines, with a preference for GitHub Actions.

• Hands-on experience with Terraform for managing Azure and Databricks resources.

• Production experience with Spark/PySpark and SQL is required.

• Familiarity with batch and incremental ingestion, schema evolution, medallion data modeling, and data quality validation.

• Practical experience with Delta Lake, Unity Catalog, Jobs, and Delta Live Tables / Lakeflow.

• Working knowledge of Databricks Asset Bundles, system tables, ADLS Gen2, Entra ID, Key Vault, and cloud networking.

• Experience in supporting production systems in a DevOps, SRE, or platform capacity.

• Knowledge of monitoring and alerting, incident response, on-call duties, root cause analysis, runbooks, and disaster recovery planning.

• Strong communication and documentation skills.

• Ability to collaborate effectively in a remote team environment with both technical and business stakeholders.

• Bachelor's degree or equivalent practical experience is required.

• Knowledge of healthcare payer data formats such as X12 834/837, HL7, and HIPAA is advantageous; HITRUST experience is a plus.

• Databricks or Azure certifications are preferred.

🏝️ Benefits

• CVS Health offers a bonus, commission, or short-term incentive program in addition to base pay.

• Medical coverage is provided.

• Dental coverage is included.

• Vision coverage is available.

• Paid time off is offered.

• Various retirement savings options are accessible.

• Participation in wellness programs is encouraged.

• Additional resources supporting physical, emotional, and financial well-being are available based on eligibility.