Staff Data Engineer, Observability Data Lake

This is a fully remote position, open to applicants in Texas.

📋 Description

• Design, create, implement, and maintain scalable data pipelines and lakehouse solutions utilizing Databricks.

• Ingest, transform, and manage observability, infrastructure, and security data from various enterprise sources.

• Develop and enhance PySpark workloads, Structured Streaming jobs, and bronze/silver/gold data architectures.

• Ensure optimal performance, reliability, scalability, data integrity, and operational excellence.

• Collaborate with Security, Infrastructure, Observability, and Platform Engineering teams to integrate new data sources.

• Establish governance standards, enhance metadata management, and deliver reliable enterprise data products.

• Contribute to engineering standards, conduct code reviews, implement CI/CD practices, develop monitoring strategies, and optimize platforms.

• Identify and mitigate technical debt, process deficiencies, and operational risks.

• Advocate for best practices regarding Databricks, Delta Lake, Unity Catalog, data governance, security measures, access management, and cost-effective data platform operations.

⛳️ Requirements

• Over 7 years of experience in building and supporting enterprise data engineering solutions utilizing Python, Spark, PySpark, SQL, and distributed data processing technologies.

• More than 6 years of experience in designing, developing, and managing large-scale data pipelines, data ingestion frameworks, and bronze/silver/gold lakehouse architectures across multiple data sources.

• At least 5 years of experience working with Databricks in production settings, including Delta Lake, Unity Catalog, Databricks Asset Bundles, serverless SQL, performance enhancement, and platform management.

• A minimum of 4 years of experience in developing high-volume batch and streaming data solutions, enforcing data quality controls, troubleshooting pipeline challenges, and enhancing reliability, scalability, and operational efficiency.

• Over 2 years of experience collaborating with engineering teams, engaging in code reviews, contributing to technical standards, applying cloud security and access management principles, and communicating technical solutions to both technical and non-technical audiences.

• Experience in building Structured Streaming and near real-time ingestion solutions on Databricks or similar cloud-native data platforms.

• Familiarity with implementing Databricks governance capabilities, such as Unity Catalog, storage credentials, external locations, catalog management, and access control frameworks.

• Knowledge of OCSF or comparable observability, cybersecurity, or security event normalization frameworks.

• Experience with CI/CD pipelines for data engineering solutions using Databricks Asset Bundles, GitHub Actions, Azure DevOps, or similar automation frameworks.

• Previous experience supporting observability, infrastructure, cybersecurity, logging, telemetry, or operational analytics platforms in large-scale enterprise environments.

• Bachelor’s degree from an accredited institution or equivalent professional experience (High School Diploma/GED plus 4 years of relevant experience).

🏝️ Benefits

• CVS Health bonus, commission, or short-term incentive program.

• Target awards in the company’s equity award program.

• Medical coverage.

• Dental coverage.

• Vision coverage.

• Paid time off.

• Retirement savings options.

• Wellness programs.

• Additional resources supporting physical, emotional, and financial well-being.