Senior Data Engineer – Platform

This is a fully remote position, open to applicants in California, +3 more states.

📋 Description

• Take ownership of systems from understanding ambiguous customer and operational requirements to architecture, implementation, deployment, observability, incident response, and ongoing support.

• Design and manage batch and streaming ingestion, transformation, reconciliation, and serving pathways for fleet, capacity, utilization, cost, scheduling, and operational telemetry.

• Develop shared libraries, workflow and Directed Acyclic Graph (DAG) abstractions, deployment tooling, data contracts, and standardized platform patterns.

• Engineer dependable distributed workloads and troubleshoot correctness and performance issues across applications, SQL engines, Spark jobs, storage systems, networks, and cloud services.

• Build systems that support retries, idempotency, backfills, schema evolution, and partial failure handling.

• Implement least privilege, service identities, secrets management, access controls, environment isolation, auditability, and safe operational practices.

• Set up automated tests, data-quality checks, lineage, freshness and completeness monitoring, actionable alerting, Service Level Objectives (SLOs), and clear ownership.

• Transform trusted data into usable formats through well-structured tables, APIs, automation, dashboards, and targeted internal applications.

• Lead build reviews, communicate trade-offs, mentor engineers, and enhance architecture, testing, debugging, and operational methodologies.

⛳️ Requirements

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related discipline, or equivalent experience.

• Over 5 years of experience in building and operating production software, data platforms, backend infrastructure, databases, or distributed systems.

• Proficient in Python or another backend or systems language, with a strong preference for working primarily in Python and SQL.

• Practical experience in distributed data processing, database architecture and operation at scale, production ETL, change-data-capture, streaming/event-processing systems, backend or cloud-platform systems, or strong SQL and data modeling.

• Ability to debug unfamiliar systems across multiple layers using logs, metrics, traces, query plans, profiles, and controlled experiments.

• Experience managing services or pipelines in a cloud or complex production setting, including testing, CI/CD, monitoring, alerting, rollback, and incident response.

• Familiarity with secure platform development, identity and access management, least privilege principles, secret handling, trust boundaries, and safe multi-environment deployments.

• Capability to make architectural trade-offs, navigate ambiguity, and communicate effectively with users, partner teams, and engineers from various disciplines.

• Proven track record of learning new technologies and domains and translating that knowledge into maintainable systems and reusable team practices.

• Experience with AI agents and LLM-supported workflow automation.

• Preferred experience with Databricks, Apache Spark, PySpark, Spark SQL, Delta Lake, Unity Catalog, Kafka, change-data capture, event systems, Elasticsearch/OpenSearch, AWS, Azure, GCP, Kubernetes, Slurm, compute clusters, GPU infrastructure, fleet-scale telemetry, agentic systems, LLM-enabled workflow automation, harness engineering, or AI-agent evaluation and operational tools.

🏝️ Benefits

• Equity

• Benefits