Lead Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee the design, development, and implementation of key platform components, such as the Constellation Lakehouse in Databricks, Connect API (GraphQL) layer, and applications supported by Lakebase.

• Construct and sustain scalable batch and incremental data pipelines utilizing Databricks, Apache Spark, PySpark, SQL, and Delta Lake.

• Integrate data from transactional databases, APIs, files, and external partners into governed datasets.

• Design and oversee unified healthcare data models that encompass membership, providers, claims, encounters, clinical data, and various business domains.

• Develop ingestion and transformation processes, including change data capture, schema evolution, deduplication, historical tracking, and reconciliation.

• Build and enhance data access layers and APIs in collaboration with software engineering teams.

• Develop and support applications backed by Lakebase and PostgreSQL data structures.

• Maintain and optimize Databricks workspaces, compute resources, job orchestration, and environment settings.

• Implement data governance and access controls through Unity Catalog.

• Establish automated data quality checks, reconciliation, freshness monitoring, and alerting mechanisms.

• Apply DevOps methodologies, including Git, automated testing, CI/CD, infrastructure as code, and repeatable deployment strategies.

• Optimize pipelines, Spark workloads, SQL queries, and compute settings for performance, scalability, and cost-effectiveness.

• Investigate and address platform incidents and pipeline failures; conduct root-cause analyses.

• Collaborate on backup, recovery, and business continuity procedures; maintain documentation and runbooks.

• Perform code and design reviews, set engineering standards, and mentor engineers.

⛳️ Requirements

• Bachelor’s degree in Computer Science, Engineering, Data Science, or a related discipline.

• Over 5 years of experience in data engineering, specifically in developing and supporting enterprise data platforms and pipelines.

• At least 3 years of practical experience with Databricks in production settings.

• Strong expertise in Python, PySpark, SQL, and Apache Spark.

• Familiarity with Delta Lake and contemporary lakehouse architectures.

• Practical experience with Azure data services and cloud-based data platforms.

• Solid understanding of data modeling, relational databases, and ETL/ELT best practices.

• Experience in integrating enterprise databases and APIs into scalable data solutions.

• Proficiency with Git, CI/CD pipelines, and automated deployment methods.

• Proven capability to monitor, troubleshoot, and enhance data pipelines and platform performance.

• Demonstrated leadership in solution design, code reviews, mentoring, and cross-functional teamwork.

• Strong problem-solving abilities and effective communication skills.

• Preferred: Experience administering and optimizing Databricks environments, including Unity Catalog.

• Preferred: Familiarity with PostgreSQL, GraphQL APIs, and operational applications leveraging lakehouse data.

• Preferred: Experience with healthcare data, including claims, eligibility, provider, or clinical datasets.

• Preferred: Knowledge of HL7, FHIR, and X12 standards.

• Preferred: Experience with Terraform or other infrastructure-as-code tools.

• Preferred: Exposure to streaming data pipelines and event-driven architectures.

• Preferred: Experience in supporting AI, machine learning, or NLP data workloads.

• Preferred: Familiarity with regulated data environments and governance best practices.

• Preferred: Experience with cloud data platform migrations and modernization projects.

• Preferred: Certifications in Azure, Databricks, or AWS.

🏝️ Benefits

• Remote position available within the U.S.

• An Equal Employment Opportunity and Affirmative Action employer.

• Support for disability accommodations during the application process.