Staff Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Act as the Directly Responsible Individual (DRI) for intricate, multi-sprint platform projects, including Fivetran connector development, Databricks Lakehouse migration efforts, event streaming infrastructure, lower environment implementation, and the adoption of engineering standards.

• Design, construct, and sustain production-quality ingestion pipelines and platform infrastructure—from source connectivity through the Bronze/Silver layers—that are utilized daily by Analytics Engineering, Data Science, and business teams.

• Create, implement, and manage event-driven and streaming data pipelines utilizing Kafka, PySpark, and Databricks Structured Streaming, which encompasses defining scaling strategies, cost guardrails, consumer lag alerts, and runbooks prior to production deployment.

• Manage the ingestion and raw-to-cleansed layer (Bronze to Silver) data contracts, oversee schema governance, and establish Service Level Agreements (SLAs).

• Ensure data quality for the pipelines you develop: write dbt tests, integrate anomaly detection, validate schemas, and notify on data drift—pipelines are delivered with quality gates in place from the start.

• Ensure the reliability of the systems you create: define Key Performance Indicators (KPIs) and Service Level Objectives (SLOs), implement Datadog monitoring and alerting as code, participate in the on-call rotation, and manage Tier 1 operational tickets and runbooks for the systems within your purview.

• Oversee the integration and data activation layer, which includes Fivetran connectors and Hightouch reverse ETL pipeline connectors, managing the entire process from Infrastructure as Code (IaC) provisioning to production monitoring and schema change governance.

• Assist Analytics Engineers, Data Scientists, and Machine Learning engineers by developing platform capabilities and data pipelines that facilitate their roadmaps; collaborate with legal, security, and DevOps on compliance controls and IaC improvements as necessary. The responsibility of Data Engineering lies in the platform layer and data delivery, while transformation logic and model readiness for serving are handled by Analytics Engineering.

• Identify and address systemic inefficiencies across DPE-owned pipelines and infrastructure—focus on root causes rather than just symptoms.

• Mentor Senior Data Engineers through design reviews, code reviews, and collaborative work; support their transition from squad-level responsibilities to broader cross-squad contributions.

• Contribute to and promote the adoption of engineering standards such as testing practices, CI/CD patterns, observability-as-code, and Schema Registry governance, and engage in Architectural Review Committee (ARC) reviews for changes impacting cross-team collaboration or costs.

⛳️ Requirements

• A minimum of 8 years of professional experience in designing, building, and managing data pipelines and platform infrastructure.

• Proficiency in Change Data Capture (CDC) patterns for real-time data ingestion.

• Experience with Flink for stream processing tasks.

• Knowledge of governing and managing dbt within a production BigQuery or Databricks setting, including CI/CD configuration, testing standards, documentation standards, and platform-level schema governance. Practical dbt experience for ingestion-layer (Bronze/Silver) pipelines is essential.

• Proven experience in developing and managing Airflow DAGs at scale, focusing on task-level orchestration patterns, DAG reliability, and multi-priority scheduling.

• Demonstrated ability to build event streaming pipelines with Kafka or Confluent Kafka, including producers, consumers, schema evolution, Schema Registry governance, and consumer lag management.

• Multi-cloud proficiency across GCP and AWS, as both platforms are utilized daily: BigQuery operates on GCP, while Airflow runs on AWS EKS.

• Experience taking ownership of data quality for production pipelines, including dbt tests, anomaly detection, and alerts for schema changes and data drift.

• Familiarity with Fivetran or a similar connector platform, including IaC provisioning, schema change management, and connector health monitoring.

• Experience with the Databricks platform, including Delta Lake, Databricks Workflows, and Unity Catalog.

• Understanding of data compliance in regulated environments, encompassing HIPAA/PHI management, access controls, and audit logging.

• Expertise in Infrastructure-as-Code, preferably with Terraform or similar tools, treating infrastructure modifications as code changes.

• Strong skills in Python and SQL, with the ability to write, review, and elevate production-grade pipeline code.

• Excellent design instincts: you can interpret ambiguous requirements, create precise solution designs, and deliver to production with minimal rework.

🏝️ Benefits

• Competitive salary and equity compensation for full-time positions.

• Unlimited paid time off (PTO), company holidays, and quarterly mental health days.

• Comprehensive health benefits, including medical, dental, vision, and parental leave.

• Employee Stock Purchase Program (ESPP).

• 401(k) benefits with employer matching contributions.

• Team retreats held offsite.