Senior Data Engineer – Databricks, Unity Catalog

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, enhance, and sustain data transformation pipelines utilizing dbt Core and Databricks (Unity Catalog, Delta Lake, Spark SQL)

• Implement dimensional modeling (Star Schema) to facilitate efficient usage in BI tools

• Guarantee data quality and governance by developing both generic and specific tests in dbt, alongside conducting thorough source-to-target validations

• Operate independently from business rules specifications, identifying discrepancies and recommending effective technical solutions

• Manage code and pipelines using Git workflows, Merge Requests, and CI/CD principles through GitLab

• Develop and upkeep auxiliary Python scripts and provide assistance with understanding orchestration DAGs

⛳️ Requirements

• Advanced SQL: Expertise in CTEs, window functions, deduplication methods, data management (dates/numbers), and support for Slowly Changing Dimensions (SCD Type 2)

• dbt Core in Production: Practical experience with models, sources, tests, seeds, snapshots, Jinja macros, materializations (ephemeral, incremental), contracts, and utilization of the ref/var functions

• Databricks: Proficient in Unity Catalog, Delta Lake, SQL Warehouse, and Spark SQL

• Data Modeling: Background in dimensional modeling (Star Schema) aimed at high performance in Power BI

• Version Control & Scripting: Skilled in Git, Merge Request workflows, familiar with CI/CD in GitLab, and competent Python abilities for automation and DAG interpretation

• Technical English at a minimum B2 level for the reading and writing of technical documentation

• Preferred: 5 or more years of relevant experience in Data Engineering

• Preferred: Experience in orchestration using Apache Airflow, particularly AWS MWAA, and the Cosmos library for dbt execution

• Preferred: Familiarity with SAP ECC data, PM and CO modules, or BODS pipelines

• Preferred: Understanding of AWS architecture (S3) and Lakehouse Federation

🏝️ Benefits

• Meal allowance or meal voucher

• Discounts on courses, universities, and language institutions

• Stefanini Academy — a platform providing free, updated online courses with certificates

• Mentoring

• Benefits club for consultations and medical examinations

• Health insurance

• Dental insurance

• Benefits and discounts at leading establishments

• Travel club

• Pet insurance partnership