This is a fully remote position, open to applicants in United States.

• Create and implement high-performance, distributed web scrapers utilizing Python and Scrapy

• Employ Browser Scripting tools to navigate, interact with, and extract information from websites

• Deploy, scale, and oversee scraping tasks on Kubernetes

• Establish rigorous JSON Schemas and make use of Pydantic for data validation

• Develop and enhance search and storage pipelines using Elasticsearch

• Design robust pipeline workflows to manage the complete data lifecycle

• Handle complex proxy rotation and session management to ensure high success rates

• Over 7 years of professional experience in software engineering

• Exceptional reverse-engineering abilities

• Advanced proficiency in Python

• Extensive experience with Scrapy

• Demonstrated experience with browser automation tools

• Expertise in JSON and data validation utilizing Pydantic

• Practical experience in indexing, querying, and optimizing Elasticsearch clusters

• Strong skills in managing and scaling applications in Kubernetes environments

• Experience in constructing structured pipeline workflows

• Bachelor’s degree in Computer Science or Engineering

• Medical, dental, and vision insurance

• Life and disability insurance

• Retirement benefits

• Paid leave

• Tuition assistance

• Professional development

Data Collection Engineer

People also viewed