Senior Data Engineer, Platform

This is a fully remote position, open to applicants in United States.

📋 Description

• Take charge of the infrastructure and platform that underpins data operations, encompassing compute options, orchestration, deployment, connectivity, reliability, and observability.

• Define the technical vision, make architectural choices, compose design documentation, and decide between building or buying solutions.

• Develop the data platform roadmap, oversee delivery, and communicate updates, risks, and trade-offs effectively.

• Establish standards for design and code reviews, mentor engineers, and facilitate contributions from platform collaborators.

• Manage the architecture for Databricks and AWS platforms, focusing on ingestion, storage, compute, access control, and environments.

• Create tested, versioned connections to payer claims, eligibility, attribution, clinical, event, and care-platform systems via SFTP, S3, private networking, FHIR, and HL7.

• Enhance job performance and cost efficiency by selecting appropriate compute resources, right-sizing, leveraging serverless or cluster options, and transitioning from single-machine Python to distributed computing.

• Oversee orchestration, deployments, Terraform, GitLab CI/CD, automated testing, and release practices.

• Boost reliability and observability, enabling early detection of failures and anomalies, and design resilient operations.

• Make governed data accessible for AI applications through metric definitions, Genie spaces, and MCP servers.

• Validate incoming data by checking row counts, date ranges, expected fields, and historical coverage.

• Automate routine data operations and engage in a collaborative, lightweight data operations rotation.

• Safeguard PHI through access controls, auditing, de-identification, and secure data sharing.

• Provide analytics engineering with a dependable, timely, and cost-effective platform.

• Represent data engineering in interactions with partners, vendors, and colleagues, while sharing knowledge.

⛳️ Requirements

• Generally, seven or more years of experience in building production software or data systems.

• Several years of experience managing data infrastructure or pipelines end-to-end, from design to operation.

• Hands-on experience with at least one distributed data processing engine at scale, such as Spark, Databricks, Snowflake, BigQuery, Flink, Dask, or Ray.

• Proficient in Python and capable of writing defensible SQL.

• Experience designing systems on a major cloud platform; AWS is preferred, covering storage, compute, networking, IAM, and cost management.

• Familiarity with automated testing, CI/CD, infrastructure as code, code review, observability, and incident response practices.

• Proven experience leading technical initiatives and mentoring fellow engineers.

• Skilled in troubleshooting upstream data issues until resolution.

• Daily engagement with AI coding tools and the ability to identify incorrect AI outputs using real data.

• Bonus points for familiarity with Databricks, Delta Lake, Unity Catalog, and dbt.

• Bonus points for knowledge of Databricks Genie, MCP servers, or other governed-data methodologies for AI agents.

• Healthcare or regulated-data experience, including claims, eligibility, FHIR, HL7 v2, HIPAA, or SOC 2, is a plus.

• Familiarity with workflow orchestration, event-driven ingestion, or serverless computing is advantageous.

• Experience leading or managing engineering teams, or an interest in evolving into a data engineering lead, is favorable.

• Must reside physically in the United States.

• Must indicate whether visa sponsorship will be necessary to maintain employment authorization.

🏝️ Benefits

• Extensive opportunities for professional development.

• Varied career pathways.

• A culture that prioritizes diversity, equity, inclusion, and belonging.

• Innovative payment models and a technology-driven mission in community health.