Senior Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Take ownership of the pipelines and platform that transfer Nerdy’s data from source systems into a governed lakehouse.

• Develop and manage both streaming and batch ingestion processes from Kafka and third-party APIs into Iceberg tables.

• Oversee Change Data Capture (CDC), schema evolution, and the handling of late-arriving data.

• Manage the Iceberg lakehouse on Starburst Galaxy/Trino, including aspects such as table layout, partitioning, compaction, retention, and cluster sizing.

• Create and sustain bronze, silver, and gold dbt layers.

• Implement incremental models, idempotent merges, and safe backfill processes.

• Push shared logic upstream to streamline operations for downstream consumers.

• Design dbt Cloud schedules, dependencies, retries, and alerting systems.

• Ensure production freshness SLAs are maintained and drive incidents to their root causes.

• Establish data quality tests, contracts, freshness and volume monitoring, lineage, and access controls.

• Manage Personally Identifiable Information (PII) within established privacy and security guidelines.

• Optimize queries, file layout, and cluster utilization to enhance performance and control costs.

• Oversee clusters, storage, and access management through Terraform and Continuous Integration.

• Collaborate with Analytics Engineering, Product Engineering, and business teams to create reliable data products.

• Maintain versioned Git code, review pull requests, conduct CI/CD processes, and document sources, models, and runbooks.

• Leverage AI for tasks such as code generation and review, test scaffolding, pipeline debugging, data profiling, and anomaly detection.

⛳️ Requirements

• Bachelor’s degree in Computer Science, Engineering, Math/Statistics, or equivalent experience.

• 3+ years of experience as a Data Engineer or in a similar role within high-growth environments.

• Proficient in SQL and skilled in Python.

• Experience in constructing unattended, self-recovering production pipelines.

• Familiarity with Iceberg on Trino/Starburst Galaxy or comparable lakehouse technologies such as Delta Lake, Hudi, Spark, Databricks, or Presto/Athena.

• Streaming experience with Kafka/Confluent or similar, including schema registry, CDC, and idempotent lakehouse loading.

• Proficient in dbt and dbt Cloud, including incremental models, macros, tests, and documentation.

• Experience with Airflow or Dagster.

• Strong AWS skills, particularly with S3, IAM, and Glue/REST catalogs.

• Experience with Infrastructure as Code using Terraform or similar tools.

• Knowledge of data modeling, partitioning, incremental processing, and strategies for large-dataset backfills.

• Familiarity with Git-based development and CI/CD practices.

• A disciplined approach to code reviews, release management, and on-call responsibilities.

• Experience utilizing AI-native productivity tools.

• Alignment with Nerdy’s apolitical, mission-driven culture.

🏝️ Benefits

• Competitive compensation benchmarked to USD, paid in local currency.

• Fully remote work within your home country; no relocation necessary.

• Flexible paid time off policy.

• Paid official national holidays.

• Complimentary all-inclusive learning membership for the employee and their household.

• Access to 1-on-1 tutoring hours and unlimited on-demand classes.

• Availability of Nerdy’s learning products and services.

• Access to cutting-edge AI tools.

• Regular training sessions and peer reviews.

• A collaborative organizational culture.

• Opportunity to make a global impact.