Senior Data Engineer

This is a fully remote position, open to applicants in Europe.

📋 Description

• Develop concurrent ingestion systems for provider APIs, webhooks, files, and web scraping, incorporating rate limits and error handling.

• Manage scheduled workflows and historical data backfills utilizing Airflow, Dagster, or Prefect.

• Create event-driven intake processes with Pub/Sub or Cloud Tasks, which include retry mechanisms, dead-letter queues, and controlled replay of failed tasks.

• Retain original payloads, such as provider JSON, in Google Cloud Storage prior to transforming them with dbt.

• Set up medallion layers for raw, standardized, and curated datasets.

• Implement idempotent upserts, secure retries, replays, and historical backfills to prevent duplicate entries or data corruption.

• Ensure data is traceable back to its source and safe for reprocessing.

• Establish data contracts, automated validations, pipeline tests, freshness checks, and failure monitoring systems.

• Design and enhance PostgreSQL schemas, queries, and loading methodologies.

• Integrate provider enrichment and LLM-based structured extraction from text into the platform.

⛳️ Requirements

• Proven experience in building multi-source production data platforms from the ground up, encompassing initial architecture, pipeline implementation, deployment, and ongoing maintenance.

• Experience in using workflow orchestration tools such as Airflow, Dagster, Prefect, or Temporal, including managing dependencies, retries, and recovery from failures.

• Practical experience with dbt and Google Cloud Platform (GCP).

• Background in developing event-driven systems utilizing messaging or task queues, focusing on asynchronous processing and error recovery.

• Strong proficiency in SQL and PostgreSQL, including data modeling, query optimization, and reliable incremental loading techniques.

• Familiarity with data contracts, raw and curated storage, retention strategies, and automated quality assurance checks.

• Advanced Python skills, particularly in asynchronous HTTP and concurrent processing using tools like asyncio, aiohttp, or HTTPX.

• Direct experience with web scraping tools such as Scrapy, Playwright, Selenium, or Beautiful Soup, including concurrent scraping, rate limiting, and error handling.

• Preference for candidates with Airflow experience.

• Experience in startups or small engineering teams is a plus.

• Background in entity resolution is advantageous.

• Experience with LLM-based structured extraction or enrichment of company and people data is a plus.

• Familiarity with BigQuery and Dataform is desirable.

🏝️ Benefits

• Fully remote work arrangement.

• Opportunity to take ownership of the data platform from its architectural design to production.

• Significant ownership and adaptability to shifting priorities within startup or small-team settings.