Senior Data Engineer

This is a fully remote position, open to applicants in United States, +1 more country.

📋 Description

• Design, develop, and enhance the foundational data platform infrastructure, which includes distributed query engines, orchestration, warehousing, and cataloging.

• Take ownership of the lakehouse infrastructure as code, overseeing deployments through Terraform and Ansible on Kubernetes.

• Create and sustain low-latency streaming and CDC ingestion pipelines, along with batch ingestion pathways landing in Iceberg.

• Develop and expand the BI landscape to facilitate efficient, self-service access to lakehouse data.

• Implement best practices for platform reliability, such as monitoring and alerting, on-call rotations, incident response, maintenance windows, runbooks, and SLAs.

• Collaborate with DevOps, Analytics Engineering, and other stakeholders to address infrastructure gaps and accommodate new data needs.

⛳️ Requirements

• Over 5 years of experience in Data Engineering, including at least 2 years of building and maintaining scalable, low-latency data platforms handling over 100 million events per day.

• Extensive hands-on experience managing data infrastructure on Kubernetes, utilizing cloud-native tools such as Docker and Helm.

• Experience in production environments with Infrastructure as Code (IaC) tools: Terraform, Ansible, and ArgoCD (or similar tools).

• In-depth understanding of distributed systems (including storage, transactions, and query processing) with practical experience operating open-source query engines like Trino or Presto.

• Significant experience with object storage and open table formats, particularly Apache Iceberg.

• Familiarity with streaming and CDC systems such as Kafka, Redpanda, and Debezium.

• Practical experience using orchestration frameworks (e.g., Airflow) and ELT tools (e.g., Airbyte).

• Strong proficiency in Python and SQL for developing pipelines and platform tools.

• Experience with Google Cloud Platform and its data services (GCS, Cloud Build, Cloud SQL, Dataproc, etc.); or relevant experience with other cloud services.

• Capability to excel in a fast-paced startup environment and adapt infrastructure to swiftly changing demands.

🏝️ Benefits

• Competitive Salary & Stock Options

• Health Benefits

• New Hire Home-Office Setup: One-time USD $500

• Monthly Stipend: USD $150 per month via a Brex Card