Senior Data Engineer

This is a fully remote position, open to applicants in United Kingdom.

📋 Description

• Design, develop, and manage systems that support data products, including data ingestion, processing, analytical storage, and serving.

• Create and implement scalable, high-throughput data pipelines and platform systems for blockchain data across more than 30 chains.

• Develop backend data services and loaders using Go and Python.

• Design and refine ClickHouse data models and analytical workloads, optimizing them for sub-second query performance.

• Construct incremental, idempotent Databricks pipelines managing billions of rows and tens of terabytes of data.

• Enhance platform scalability, reliability, freshness, performance, and cost-effectiveness.

• Investigate and resolve complex performance and reliability challenges.

• Collaborate with product, analytics, and investigations teams to convert requirements into effective platform designs.

• Participate in architecture discussions, design reviews, and code evaluations.

• Mentor engineers and foster strong engineering and operational practices.

• Take ownership of projects from design documentation through to production and be accountable for the results of pipelines.

⛳️ Requirements

• 5+ years of experience in building and operating production data systems, including ETL and ELT pipelines.

• Minimum of 2 years of experience managing systems that other engineers rely on.

• Proficient in Python and strong in SQL.

• Experience with typed, tested code; familiarity with asyncio and concurrency.

• Experience in building incremental data models encompassing billions of rows.

• Proven production experience with Go for backend services, high-throughput services, or data loaders.

• Hands-on experience with ClickHouse in production, including MergeTree schema and sort-key design, materialized views, bulk loads, and query optimization.

• Practical experience with Databricks in production using PySpark, Delta Lake, and Unity Catalog.

• Skilled in designing incremental and idempotent tables and monitoring compute costs.

• Extensive hands-on experience with pipelines managing billions of rows and tens of terabytes of data.

• Knowledge of partitioning, clustering, and file sizing impacts on cost and latency.

• Experience with Kafka, Airflow, and Kubernetes in production settings.

• Understanding of delivery semantics and idempotent writes.

• Familiarity with handling backfills and failures.

• Experience running data workloads using Docker and Helm.

• Preference for Google Cloud; AWS is acceptable.

• Ability to instrument systems, write runbooks and post-incident reviews, and create design documentation.

• Experience with blockchain data is advantageous but not essential.

• Familiarity with graph or distributed key-value stores is a plus.

• Experience in infrastructure as code, GitOps, observability, and load-test design is beneficial.

• Experience in building platforms that serve multiple data sources, datasets, or product teams is advantageous.

• Experience in AI-assisted development and knowledge of fintech, compliance, or AML is a plus.

🏝️ Benefits

• Comprehensive health insurance coverage.

• Flexible time off policy.

• Opportunities for learning and development.

• Work hours designed to promote work/life balance.

• Team-building activities.

• Support for mental health discussions.

• Competitive compensation package.

• Generous equity options.

• Opportunities for career advancement.