Senior Data Engineer

This is a fully remote position, open to applicants in United Kingdom, +1 more country.

📋 Description

• Develop and manage batch and real-time pipelines that support matching and search systems.

• Engage with real-time data streams and search infrastructure utilizing Kafka and Elasticsearch.

• Design datasets utilized by the product layer, encompassing candidate features, interaction events, and ranking signals, while ensuring stable contracts.

• Take ownership of production pipelines, which includes monitoring, alerting, backfills, and on-call support.

• Instrument recruiter and client behaviors to enable measurement and enhancement of matching quality.

• Construct event pipelines that support Customer Satisfaction (CSAT) and workflow metrics.

• Execute and analyze experiments to refine data-processing pipeline quality and ensure experiment data is queryable.

• Implement data-quality checks and tests at critical boundaries and enhance visibility of failures.

• Minimize relevant costs and latency.

• Collaborate with QA to ensure data correctness is verifiable.

• Identify and provide valuable internal or external data sources for structured search and Generative AI tools.

• Work closely with the Lead Search Engineer, AI/ML Engineer, and Product Manager as part of a frequently shipping team.

⛳️ Requirements

• Over 5 years of experience in building production data pipelines.

• Proficient in Python programming, with hands-on experience writing clean, production-ready code.

• Strong SQL skills and familiarity with PostgreSQL or other relational databases.

• Extensive hands-on experience with Elasticsearch / OpenSearch, including performance optimization and query tuning.

• Solid understanding of both batch and event/streaming patterns, with a clear awareness of when to apply each.

• Practical skills in data modeling for analytics and serving application features.

• Experience managing your own pipelines in production, along with comfort in working within a codebase that includes tests, code reviews, and CI.

• Familiarity with message brokers such as Kafka or RabbitMQ (preferable but not required).

• Experience with cloud data warehouses, with a preference for BigQuery and acceptance of equivalents (preferable but not required).

• Knowledge of Docker, Kubernetes, and cloud platforms like AWS and/or GCP (preferable but not required).

• Experience contributing to search, ranking, or machine learning systems specifically (preferable but not required).

• Building extraction or enrichment pipelines based on large language models (preferable but not required).

• Experience with dbt or a similar transformation framework (preferable but not required).

• Familiarity with observability tools like Prometheus or Datadog and background job processing frameworks (preferable but not required).

• Exposure to Elixir services (preferable but not required).

🏝️ Benefits

• Tenure gifts, including vouchers, additional holiday, and sabbaticals for each year of service.

• Health insurance provided through Vitality.

• Option for remote work up to 20 days each year, offering flexibility and a change of environment.

• Employee Assistance Programme that provides personalized health and wellbeing advice from specialized teams.

• Enhanced maternity and paternity leave pay.

• 25 days of annual leave in addition to bank holidays, including a week closure over Christmas.

• Access to the MyMindPal app for online mental fitness support.

• Corporate events, ranging from quarterly meet-ups to annual winter and summer celebrations.