Senior Anti-Bot/Scraping Engineer

This is a fully remote position, open to applicants in United Kingdom.

📋 Description

• Design and sustain large-scale Python scraping systems that function reliably against Google and other highly secured websites.

• Develop scrapers for dynamic, JavaScript-heavy webpages utilizing browser automation and hybrid techniques.

• Continuously adapt to alterations in page layouts, request flows, and anti-automation strategies.

• Construct robust extraction pipelines with a focus on data accuracy, consistency, and observability.

• Implement strategies for proxy usage, fingerprinting, and request routing to enhance stability and throughput.

• Monitor the health of scrapes, identify failures promptly, and troubleshoot production issues.

• Optimize performance, costs, and latency throughout the scraping infrastructure.

• Collaborate with data engineers, data scientists, and product teams to guarantee the usability and trustworthiness of scraped data.

• Document scraping methodologies and operational runbooks for sustainable maintenance.

• Report directly to the Head of Engineering.

⛳️ Requirements

• Significant professional experience with Python or another primary language such as GoLang, Rust, or JavaScript, including production-quality scraping systems.

• Extensive hands-on experience scraping Google or similarly secured platforms.

• Strong understanding of HTTP, TLS, cookies, headers, redirects, and browser networking behaviors.

• Experience with browser automation frameworks such as Playwright, Selenium, Puppeteer, or similar tools.

• In-depth knowledge of HTML parsing, DOM traversal, and data extraction methods.

• Experience managing rate limiting, CAPTCHA, IP rotation, and bot detection systems.

• Comfortable working with asynchronous and concurrent scraping architectures.

• Experience executing scrapers at scale in cloud environments.

• Strong debugging skills and the capability to analyze complex failure scenarios.

• Familiarity with headless browser fingerprinting and anti-bot evasion techniques.

• Experience with Docker and Kubernetes.

• Knowledge of distributed task systems and job queues.

• Background in data quality monitoring and anomaly detection.

• Experience with search, advertising, or competitive intelligence data.

• Ability to articulate complex technical behaviors clearly to non-specialists.

• UK work authorization is requested in the application form.

🏝️ Benefits

• Stock options.

• Trust-based vacation.

• Remote-first work environment.

• Flexible working hours.

• Career growth and tailored training through “Sherpa plans”.

• Monthly hackdays and training days.

• Up to 6 months of fully paid maternity leave.

• 2 months of paternity leave.

• $200 stipend for home office setup.

• $70 credit for welcome swag gifts.

• A day off for your birthday.

• Summer and Christmas celebrations.

• Annual and quarterly offsite events.

• Monthly meet-ups.

• Paid volunteer time each quarter.

• Startup engineering culture.