Senior Data Engineer, Observability Engineering

This is a fully remote position, open to applicants in Arizona.

📋 Description

• Design, develop, deploy, and maintain scalable data pipelines and lakehouse solutions based on Databricks.

• Ingest, transform, and organize observability, infrastructure, and security data from various enterprise sources.

• Create and enhance PySpark workloads, Structured Streaming applications, and bronze/silver/gold data architectures.

• Guarantee high performance, reliability, scalability, data quality, and operational excellence.

• Collaborate with Security, Infrastructure, Observability, and Platform Engineering teams to integrate data sources.

• Establish governance standards, enhance metadata management, and provide reliable enterprise data products.

• Contribute to engineering standards, conduct code reviews, implement CI/CD practices, develop monitoring strategies, and optimize platform performance.

• Identify and resolve technical debt, process deficiencies, and operational risks.

• Advocate for best practices pertaining to Databricks, Delta Lake, Unity Catalog, data governance, security measures, access management, and cost-effective operations.

⛳️ Requirements

• Over 5 years of experience in building and supporting enterprise data engineering solutions utilizing Python, Spark, PySpark, SQL, and distributed data processing technologies.

• More than 5 years of experience in designing, developing, and operating large-scale data pipelines, data ingestion frameworks, and bronze/silver/gold lakehouse architectures across diverse data sources.

• At least 4 years of experience working with Databricks in production settings, including Delta Lake, Unity Catalog, Databricks Asset Bundles, serverless SQL, performance optimization, and platform operations.

• A minimum of 3 years of experience in developing high-volume batch and streaming data solutions, implementing data quality checks, troubleshooting pipeline issues, and enhancing reliability, scalability, and operational efficiency.

• Over 2 years of experience collaborating with engineering teams, engaging in code reviews, contributing to technical standards, applying cloud security and access management principles, and effectively communicating technical solutions to both technical and non-technical stakeholders.

• Experience in building Structured Streaming and near real-time ingestion solutions within Databricks or similar cloud-native data platforms.

• Proficiency in implementing Databricks governance functionalities, including Unity Catalog, storage credentials, external locations, catalog management, and access control frameworks.

• Familiarity with OCSF or similar observability, cybersecurity, or security event normalization frameworks.

• Experience in establishing CI/CD pipelines for data engineering solutions using Databricks Asset Bundles, GitHub Actions, Azure DevOps, or comparable automation frameworks.

• Previous experience supporting observability, infrastructure, cybersecurity, logging, telemetry, or operational analytics platforms in large-scale enterprise settings.

• Bachelor’s degree from an accredited institution or equivalent work experience; High School Diploma/GED with 4 years of relevant experience is also acceptable.

• Ability to work from home in Arizona.

🏝️ Benefits

• CVS Health bonus, commission, or short-term incentive program in addition to base salary.

• Medical coverage.

• Dental coverage.

• Vision coverage.

• Paid time off.

• Retirement savings options.

• Wellness programs.

• Additional resources supporting physical, emotional, and financial well-being.