Member of Technical Staff – Knowledge Platform

atavraRemoteBR flagBrazilFull-timeSoftware EngineerMid-levelSenior

Posted 1 day ago

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Develop and manage crawlers, scrapers, and browser RPA for various websites and portals

• Implement scheduling, retry mechanisms, session management, and change detection

• Oversee source coverage and freshness while repairing collections when sources undergo changes

• Execute event-driven data ingestion using Kafka and Pub/Sub

• Incorporate ordering, replays, dead-letter queues, backpressure, and idempotent consumers

• Construct batch and incremental data pipelines utilizing Spark, dbt, and SQL

• Transform raw captures into organized, typed relational tables

• Manage the Parquet and Iceberg lakehouse, including schema evolution, partitioning, compaction, and snapshots

• Set standards for data quality, contracts, tests, and lineage

• Operate systems using Kubernetes with Helm, Terraform, and CI/CD

• Collaborate with Security and Legal teams on rate limits, terms of use, LGPD, and personal data management

• Evaluate freshness, source coverage, recovery time, onboarding time, pipeline reliability, data quality, and processing costs


⛳️ Requirements

• Over 5 years of pertinent engineering experience in tech and product-oriented companies

• Proven production expertise in at least one of web scraping, large-scale crawlers, or browser RPA

• Background in data engineering with extensive experience handling large data volumes using Spark, dbt, and SQL

• Familiarity with columnar and table formats like Parquet and Iceberg

• Experience with Kafka and/or Pub/Sub in a production environment

• Proficient in production coding with Python and JavaScript

• Practical experience with Kubernetes, including using Helm

• Knowledge of infrastructure as code using Terraform

• Experience in building CI/CD pipelines with reliable rollback processes

• Readiness to operate and manage what you build, including incident response

• Daily engagement with coding agents, with a thoughtful approach to their safe usage


🏝️ Benefits

• Flexible remote work options

• Opportunity to engage with cutting-edge AI and relational foundation models

• Collaborate with prominent enterprises across Latin America

• Work alongside Security and Legal teams on ethical data collection practices

• Gain exposure to AWS, Google, NVIDIA, and various cloud data services

People also viewed

Talan8 hours ago

Senior Oracle APEX Developer

US flagTexas OnlyFreelanceSoftware Engineer$110k – $130k/year
ApplyView job
Ryder System, Inc.8 hours ago

Application Developer IV

US flagOhio OnlyFull-timeSoftware Engineer$90k – $100k/year
ApplyView job
1Password8 hours ago

Senior Developer – Device Trust

US flagUnited States OnlyFull-timeSoftware Engineer$153k – $214k/year
ApplyView job
Gainwell Technologies8 hours ago

Programmer Analyst

US flagUnited States OnlyFull-timeSoftware Engineer$58.2k – $83.2k/year
ApplyView job
Chainguard8 hours ago

Developer Outreach Lead

US flagUnited States OnlyFull-timeSoftware Engineer$151k – $178k/year
ApplyView job
Fresche Solutions8 hours ago

Senior z/OS System Programmer

US flagUnited States OnlyFreelanceSoftware Engineer
ApplyView job

Never miss a great job!

Get handpicked remote jobs straight to your inbox weekly.

Trusted by 7,400+ designers