Senior Data Product Engineer

This is a fully remote position, open to applicants in Latin America.

📋 Description

• Design and scale distributed web scraping architectures to gather data from thousands of online sources.

• Create strategies to navigate obstacles such as Cloudflare, CAPTCHAs, rate limiting, IP banning, browser fingerprinting, and other anti-scraping technologies.

• Implement proxy pools, headless browsers, and adaptive crawling methods for effective large-scale data collection.

• Process and extract information from HTML and JSON formats.

• Refactor and optimize Python data collection pipelines for production use.

• Enhance reliability, observability, error management, retries, and alerting mechanisms.

• Develop schedulable and containerized data ingestion workflows utilizing Airflow or similar tools.

• Design PostgreSQL schemas, views, partitioning, and data access methodologies.

• Utilize AWS services including S3, EKS, IAM/IRSA, Parameter Store, and ECR.

• Construct and maintain the data access layer through GraphQL and Hasura.

• Collaborate with Data Science and SaaS teams to incorporate machine learning components and establish reliable API contracts.

• Take ownership of the data product from start to finish, participate in code reviews, and promote technical enhancements.

⛳️ Requirements

• Extensive hands-on experience in large-scale web scraping.

• Proven track record in overcoming anti-scraping and bot-protection measures, including proxy rotation, headless browsers, rate limiting, IP blocking, fingerprinting, or comparable challenges.

• Strong proficiency with web scraping and DOM parsing tools such as Scrapy, Playwright, Selenium, Puppeteer, BeautifulSoup, lxml, XPath, or CSS selectors.

• Significant professional background in Python and data processing techniques.

• Practical experience with AWS, especially S3 and boto3; familiarity with EKS, IAM/IRSA, Parameter Store, or ECR is highly advantageous.

• Experience in constructing dependable data pipelines using Airflow or an equivalent orchestrator.

• Solid understanding of PostgreSQL and the principles of relational databases.

• Capability to take complete ownership of technical solutions and operate independently.

• Experience in contributing to code reviews and upholding high software engineering standards.

• Familiarity with GraphQL, Hasura, Redis, Elasticsearch, Docker, or Kubernetes is a plus.

• Background in cybersecurity data, NLP/ML pipelines, or data-as-a-product environments is a plus.

• Proficiency in English at B2 level or higher, with the ability to communicate effectively with technical and cross-functional teams.

🏝️ Benefits

• Fully flexible and self-managed vacation time.

• Sick leave and personal days off.

• Observance of public holidays.

• Paternity and maternity leave available.

• Study leave provisions.

• Designated moving days.

• Training in best practices and technology.

• Access to books and informal discussions.

• In-house English language courses.

• Continuous feedback opportunities.

• One-on-one career development sessions.

• Flexible working hours.

• Provision of equipment and work materials.

• Internal events and team-building activities.

• A day off to celebrate your birthday.

• Fully remote positions available across LATAM.

• Contractor model with payment in USD.