Senior Data Engineer, Redshift, Airflow, dbt

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, construct, and sustain ingestion and transformation pipelines managed through Apache Airflow, incorporating retries, idempotency, alerting, and SLAs.

• Create and enhance dbt transformation models within the data warehouse, structuring staging, intermediate, and marts layers with thorough tests and documentation.

• Execute ingestion processes from transactional databases, third-party APIs, regulatory documents, and operational spreadsheets.

• Define and advocate for the warehouse data model, encompassing keys, grain, historization, and retroactive adjustments.

• Set standards for distribution, sorting, partitioning, and compression within MPP/Redshift environments.

• Oversee performance and cost optimization by scrutinizing query plans, queues/WLM, concurrency, VACUUM/ANALYZE, queries, and materializations.

• Implement observability and automated quality controls addressing freshness, volume, contracts, and reconciliation.

• Design data layers and contracts across teams, establishing the source of truth and exposure for BI.

• Ensure lineage, traceability, access controls, sensitivity-based segregation, and adherence to Brazil’s LGPD and industry standards.

• Document architectural choices, maintain the data dictionary, and engage in code reviews, pair programming, and standard-setting.

• Collaborate with teams in Wealth Management, Sales, Risk, Compliance, and Operations.

• Act as a technical resource for mid-level and junior engineers.

⛳️ Requirements

• Over 6 years of experience in data engineering, with at least 2 years in architectural roles.

• Demonstrated experience in deploying and managing an analytical data platform, including handling incidents, participating in on-call rotations, and understanding real-world impacts.

• Advanced SQL skills: including window functions, recursive CTEs, execution-plan analysis, and resolving data skew and disk spills.

• Proficient in Python for data engineering, producing modular, testable, version-controlled code, utilizing libraries such as pandas or polars, type annotations, and automated testing.

• Experience with Apache Airflow in a production environment: including DAG creation, sensors, backfilling, dependency management, and failure mitigation.

• Familiarity with cloud-based MPP data warehouses, ideally Redshift; experience with Snowflake, BigQuery, or Databricks is also acceptable.

• Proficient in dbt or similar version-controlled transformation tools, with a focus on testing.

• Knowledge of dimensional modeling (Kimball), covering fact and dimension tables, grain, SCD Types 1 and 2, bridge tables, and snapshots.

• Experience with Git and a code review culture, applying CI/CD methodologies to data.

• Skills in performance diagnosis, including the ability to explain slow queries and validate fixes with quantifiable outcomes.

• Preferred: experience in financial services, regulatory data, Terraform, AWS services, Iceberg, Delta, lakehouse architectures, Kafka, Debezium, Kinesis, data quality and catalog tools, BI, and significant migrations.

• Strong numerical accuracy, effective written communication, autonomy, technical leadership, mentorship, and the capability to translate business requirements into sustainable data designs.

🏝️ Benefits

• Ongoing professional development.

• A dynamic and collaborative work environment.

• Leega is committed to being an inclusive workplace for all.

• Position is also open to candidates with disabilities.