
Member of Technical Staff – Knowledge Platform
Posted 1 day ago

Posted 1 day ago
This is a fully remote position, open to applicants in Brazil.
• Develop and manage crawlers, scrapers, and browser RPA for various websites and portals
• Implement scheduling, retry mechanisms, session management, and change detection
• Oversee source coverage and freshness while repairing collections when sources undergo changes
• Execute event-driven data ingestion using Kafka and Pub/Sub
• Incorporate ordering, replays, dead-letter queues, backpressure, and idempotent consumers
• Construct batch and incremental data pipelines utilizing Spark, dbt, and SQL
• Transform raw captures into organized, typed relational tables
• Manage the Parquet and Iceberg lakehouse, including schema evolution, partitioning, compaction, and snapshots
• Set standards for data quality, contracts, tests, and lineage
• Operate systems using Kubernetes with Helm, Terraform, and CI/CD
• Collaborate with Security and Legal teams on rate limits, terms of use, LGPD, and personal data management
• Evaluate freshness, source coverage, recovery time, onboarding time, pipeline reliability, data quality, and processing costs
• Over 5 years of pertinent engineering experience in tech and product-oriented companies
• Proven production expertise in at least one of web scraping, large-scale crawlers, or browser RPA
• Background in data engineering with extensive experience handling large data volumes using Spark, dbt, and SQL
• Familiarity with columnar and table formats like Parquet and Iceberg
• Experience with Kafka and/or Pub/Sub in a production environment
• Proficient in production coding with Python and JavaScript
• Practical experience with Kubernetes, including using Helm
• Knowledge of infrastructure as code using Terraform
• Experience in building CI/CD pipelines with reliable rollback processes
• Readiness to operate and manage what you build, including incident response
• Daily engagement with coding agents, with a thoughtful approach to their safe usage
• Flexible remote work options
• Opportunity to engage with cutting-edge AI and relational foundation models
• Collaborate with prominent enterprises across Latin America
• Work alongside Security and Legal teams on ethical data collection practices
• Gain exposure to AWS, Google, NVIDIA, and various cloud data services
Talan
Ryder System, Inc.
1Password
Gainwell Technologies
Get handpicked remote jobs straight to your inbox weekly.