Research Engineer – Web Crawlers

This is a fully remote position, open to applicants in United Kingdom.

📋 Description

• Develop and manage extensive distributed web crawlers that discover, retrieve, and extract data from billions of web pages.

• Address challenges related to content extraction from complex HTML, implement web-scale deduplication, and devise strategies for freshness and recrawling, along with ensuring politeness and handling rate limits.

• Design specialized crawling pipelines for valuable audio, video, and multilingual data sources.

• Transform crawled data into clean datasets that are ready for training purposes.

• Build tools and infrastructure that allow researchers to request, monitor, and analyze newly crawled web data.

• Take ownership of crawling systems that gather large-scale, high-quality data from the open web to support cutting-edge AI models.

⛳️ Requirements

• No formal certifications or degrees are necessary.

• Ability to showcase the resolution of significantly challenging problems through previous projects, designs, or contributions on GitHub.

• Practical experience in constructing and scaling web crawlers or scraping systems, particularly for machine learning training datasets.

• Strong expertise in distributed systems engineering at scale.

• Familiarity with Kubernetes, queue-based architectures, or custom pipelines that process billions of documents.

• Capability to independently assess the quality, coverage, and compliance of crawled data.

• Proficient in developing tools to evaluate the quality, coverage, and compliance of crawled data.

🏝️ Benefits

• Annual discretionary stipend for professional development.

• Annual discretionary fund for social travel to connect with colleagues.

• Annual company offsite event.

• Monthly co-working stipend for employees who are not located near a main hub.

• Flexibility for remote work.

• Opportunity to work from ElevenLabs offices located in London, New York, San Francisco, or Warsaw.

• Engaging, high-impact environment focused on AI research and product development.

• Opportunities for growth beyond the immediate role and responsibilities.