Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Reconstruct legacy Data Warehouse pipelines in Databricks utilizing PySpark and Spark SQL;

• Establish bronze, silver, and gold layers in accordance with the medallion architecture framework;

• Implement CDC and batch ingestion methodologies as per project specifications;

• Conduct reconstruction phases by domain, operating alongside the legacy Data Warehouse until transition;

• Apply business rules and transformations with the aid of automated testing;

• Carry out reconciliation and parity validation between data in the legacy environment and the Lakehouse;

• Enhance pipeline performance and reduce costs through partitioning, OPTIMIZE, Z-ORDER, and workload sizing;

• Add to technical documentation for migrated rules;

• Assist in the prioritization and execution of migration phases;

⛳️ Requirements

• Proficient in PySpark and Python;

• Experience in developing large-scale batch processing pipelines;

• Advanced skills in SQL;

• Familiarity with relational and dimensional data modeling;

• Practical experience with Databricks and Delta Lake;

• Understanding of the medallion architecture (bronze, silver, and gold);

• Proficient in Delta Live Tables, Lakeflow, and Asset Bundles;

• Experience in migrating or reconstructing ETL pipelines;

• Ability to translate business rules from legacy systems to Spark;

• Knowledge of CDC and batch ingestion techniques from relational databases;

• Experience with AWS (S3, Glue, EMR, Athena, Lambda, DMS, and Step Functions);

• Familiarity with Azure Synapse (SQL Pools and Pipelines);

• Experience in data quality assurance and reconciliation;

• Knowledge of automated testing and Data Quality measures;

• Proficient in Git and CI/CD practices applied to data engineering;

🏝️ Benefits

• Multi-benefits card – you have the flexibility to choose how and where to utilize it.

• Educational scholarships for Undergraduate, Graduate, MBA, and Language courses.

• Certification incentive programs.

• Flexible working hours.

• Competitive salary packages.

• Annual performance evaluations accompanied by a structured career development plan.

• Opportunities for international career advancement.

• Wellhub and TotalPass.

• Private pension scheme.

• Assistance with childcare.

• Medical insurance coverage.

• Dental insurance coverage.

• Life insurance coverage.