Staff Data Engineer, Azure Databricks Platform

This is a fully remote position, open to applicants in Illinois.

📋 Description

• Manage the operational integrity of the Azure Databricks platform.

• Enhance and maintain solutions for monitoring, logging, and alerting.

• Lead incident responses and conduct root cause analyses.

• Improve and maintain essential Azure Databricks components.

• Establish the strategic roadmap for the platform while adopting new features, services, and best practices.

• Implement and strengthen data security and governance policies.

• Oversee Unity Catalog for precise access control, data lineage, and compliance with regulations.

• Execute cost management strategies and optimize performance at the platform level.

• Streamline Databricks clusters and workloads.

• Manage automated provisioning, scaling, and resource oversight using Terraform and GitHub CI/CD pipelines.

• Mentor data engineers and Azure Databricks platform engineers.

• Create reusable patterns and libraries while offering technical support and training.

• Collaborate with data engineering and business teams to ascertain platform requirements.

• Work alongside security, networking, and infrastructure teams.

• Supervise and guide 2 platform engineers.

⛳️ Requirements

• A minimum of 8 years in a hands-on engineering position, concentrating on cloud data platforms, distributed systems, and data engineering.

• Extensive hands-on experience with Azure Databricks, encompassing administration, cluster management, job orchestration, Delta Lake, and Unity Catalog.

• In-depth knowledge of Azure, including Azure Data Lake Storage (ADLS), Azure Active Directory (AAD), and networking components.

• Strong skills in Terraform, Python, Apache Spark/PySpark, SQL, and Shell scripting.

• Proven track record in Infrastructure as Code and automated CI/CD pipelines.

• Comprehensive understanding of distributed computing concepts and large-scale data systems.

• Excellent analytical and problem-solving abilities.

• Outstanding communication, documentation, and presentation skills.

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related discipline.

• Relevant certifications such as Microsoft Certified: Azure Solutions Architect Expert, Databricks Certified Data Engineer Professional, and Databricks Certified Platform Administrator.

• Experience with Azure Synapse, Kafka, or Snowflake is preferred.

• Familiarity with healthcare data and insurance feeds is preferred.

🏝️ Benefits

• CVS Health bonus, commission, or short-term incentive program.

• Equity award program.

• Medical coverage.

• Dental coverage.

• Vision coverage.

• Paid time off.

• Retirement savings options.

• Wellness programs.

• Additional resources supporting physical, emotional, and financial well-being.