Anti-Bot Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• Design, develop, and sustain extensive Python-based web scraping systems aimed at highly secured websites (including environments similar to Google).

• Create resilient extraction frameworks capable of managing dynamic, JavaScript-heavy pages through browser automation and hybrid methodologies.

• Continuously modify systems in response to frequent alterations in page structures, request flows, and anti-bot strategies.

• Construct robust, production-ready data extraction pipelines with a strong focus on accuracy, observability, and fault tolerance.

• Apply advanced techniques such as proxy rotation, fingerprinting, session management, and request routing to maintain stability at scale.

• Monitor the health of systems, proactively identify anomalies, and troubleshoot complex production failures across distributed environments.

• Enhance scraping infrastructure for improved performance, cost-effectiveness, and reliability at scale.

• Collaborate with data engineering and product teams to ensure that the scraped data is structured, validated, and reliable.

• Operate and enhance systems that run continuously in cloud environments.

• Document system architectures, scraping methodologies, and operational procedures to ensure long-term maintainability.

• Contribute to enhancing resilience, automation, and adaptability in challenging environments.

⛳️ Requirements

• Over 7 years of professional experience in software engineering, with a strong emphasis on backend systems, data engineering, or distributed systems.

• Demonstrated experience in building and managing large-scale production web scraping systems.

• Extensive hands-on experience scraping Google or similarly intricate, heavily protected/anti-bot environments.

• Strong proficiency in Python (or similar production languages such as Go, Rust, or JavaScript).

• Comprehensive understanding of HTTP internals: headers, cookies, TLS, redirects, sessions, and browser networking behavior.

• Familiarity with browser automation frameworks such as Playwright, Selenium, Puppeteer, or equivalent.

• In-depth knowledge of HTML parsing, DOM traversal, and high-performance data extraction methodologies.

• Proven experience in managing anti-bot systems, including rate limiting, CAPTCHAs, IP rotation, and fingerprinting.

• Experience in designing asynchronous and concurrent systems for high-throughput workloads.

• Exceptional debugging skills across distributed, failure-prone production environments.

• Experience running cloud-based systems at scale with strong operational responsibility.

• Ability to work comfortably in highly adversarial and rapidly changing technical settings.

🏝️ Benefits

• Fixed Shifts: 12:00 PM - 9:30 PM IST (Summer) | 1:00 PM - 10:30 PM IST (Winter)

• No Weekend Work: Genuine work-life balance, not just a slogan

• Day 1 Benefits: Laptop and comprehensive medical insurance provided

• Support That Matters: Mentorship, community, and forums for sharing ideas

• True Belonging: A long-term career where your contributions are appreciated