Senior ML Backend Engineer – Search & Recommendations

This is a fully remote position, open to applicants in Saudi Arabia.

📋 Description

• Develop and manage query and recommendation serving functionalities, including retrieval, score fusion, filtering, and caching.

• Ensure that search-as-you-type maintains a p95 latency in the double-digit milliseconds.

• Design systems for graceful degradation in cases of slow or unavailable models and vector paths.

• Create infrastructure for experiments, encompassing shadow traffic, A/B assignment, guardrail metrics, and rollback procedures.

• Implement change-data-capture from source systems into search indexes and feature stores, ensuring measured freshness.

• Keep embeddings current for new and updated items.

• Conduct zero-downtime reindexing for collections containing millions of documents.

• Establish feature logging that transforms production traffic into training data for ranking models.

• Deploy updates to product teams' backends and clients, which includes new endpoints, cutovers, and experiment flags.

• Collaborate with product engineering teams on reviews, release schedules, and production ownership.

• Provide monitoring, alerting, runbooks, and on-call support for deployed systems.

• Work alongside search data scientists and the ranking lead on search and recommendation serving.

⛳️ Requirements

• Over 5 years of experience in building backend or platform systems in a production environment.

• More than 2 years of experience operating a production search engine (such as Elasticsearch, OpenSearch, Solr, Typesense, Vespa, or similar), including index lifecycle management, analyzers, reindexing at scale, and troubleshooting relevance issues in collaboration with data scientists.

• Proven ability to maintain low-latency services under real traffic conditions, with a clear understanding of the factors affecting p95 latency.

• Experience with streaming ingestion technologies, including Kafka, Debezium, or similar change-data-capture methods, along with idempotent backfilling.

• Proficient in Python programming.

• Comfortable with deploying changes using TypeScript.

• Experience in implementing changes through A/B testing and staged rollouts.

• Regular user of agentic coding tools such as Claude Code.

🏝️ Benefits

• Competitive salary and comprehensive benefits package.

• Opportunities for professional development and growth.

• Flexible working hours and remote work options.

• Collaborative and innovative work environment.