Senior Engineer, Ingestion

This is a fully remote position, open to applicants in United Kingdom.

📋 Description

• Take ownership of complete data collection, processing, and storage across integrations.

• Design event workflows, including queuing, retries, failure recovery, and ensuring data consistency at scale.

• Handle millions of messages while prioritizing throughput, latency, deduplication, backfills, schema design, and data-store performance.

• Develop pipeline observability and monitor operational health, data freshness, and quality.

• Collaborate with product and AI teams to pinpoint pipeline bottlenecks and implement continuous incremental updates.

• Integrate new platforms such as Discord, Reddit, Steam, X, and customer-specific sources.

• Create systems that enable rapid and cost-effective new integrations, increasingly utilizing AI agents.

• Assist in the gradual transition of ingestion systems toward an event-driven architecture.

• Make engineering trade-offs between simplification, core foundations, refactoring, and rapid adaptability.

• Provide technical clarity and drive progress in areas with ambiguous challenges.

⛳️ Requirements

• Experienced Senior Engineer with responsibility for high-volume production data pipelines.

• Capability to articulate past systems owned, detailing throughput volume, load patterns, performance bottlenecks, and structural optimizations.

• Practical experience with messaging queues, pub/sub systems, or streaming technologies.

• Understanding of ordering, idempotency, retries, and dead-letter processing.

• Proactive in risk management and accountable for maintaining operational health in production environments.

• Strong backend engineering expertise with a significant emphasis on data pipelines and infrastructure.

• Proficient in TypeScript and Node.js for backend services and database management.

• Experience in AI-native engineering utilizing coding agents.

• Strong skills in data-stores at scale, including schema design, query optimization, indexing, and partitioning.

• Familiarity with relational and analytical data stores like ClickHouse.

• Direct experience managing rate limits, schema changes, backfills, and partial failures.

• Commitment to pipeline observability, data freshness, and stringent data quality standards.

• Ability to navigate uncertainty, establish technical clarity, and drive independent progress.

• Preference for incremental delivery based on real-world usage observations.

• Clear communication skills demonstrated through concise pull requests, technical documentation, and asynchronous updates.

• Must be located within European time zones with core overlap during UK business hours.

• Nice to have: experience with GCP Pub/Sub, Kafka, or similar event-streaming technologies.

• Nice to have: familiarity with Temporal or other durable execution frameworks.

• Nice to have: hands-on experience with third-party platform APIs, rate limits, and dynamic payloads.

• Nice to have: direct experience managing datasets from Discord, Reddit, or other social media.

• Nice to have: exposure to ingested data feeding downstream AI or search systems.

• Nice to have: background in high-growth startup settings.

• Nice to have: experience migrating live systems towards event-driven architectures without service interruption.

• Nice to have: impactful technical writing, contributions to open-source projects, or engineering documentation.

🏝️ Benefits

• Highly competitive base salary - complete details provided during the initial interview.

• 25 days of annual leave plus UK public holidays.

• Options for remote or hybrid working arrangements.

• Flexible working hours.

• Access to co-working spaces, including Soho Works locations in London and the US.

• Stock option plan available.

• Home office budget granted after successful completion of probation.

• Mental health support provided through Spill.

• Biannual company meet-ups.