Data Engineer – AWS, Databricks

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Develop and sustain data ingestion, transformation, and curation pipelines;

• Create ELT/ETL pipelines utilizing PySpark and SQL throughout the Medallion layers;

• Establish data quality regulations, monitoring, and remediation protocols;

• Conduct data profiling and maintain source inventory for evaluation and governance;

• Catalog data sets and enforce data handling policies that comply with the LGPD (Brazilian Data Protection Law) using Unity Catalog;

• Engage in API integration, manage multiple data sources, and oversee orchestration processes.

⛳️ Requirements

• Extensive experience as a Data Engineer;

• Proficiency in Databricks (Delta Lake);

• Expertise in PySpark and SQL;

• Experience in constructing ELT/ETL pipelines;

• Familiarity with Medallion architecture;

• Knowledge of batch and streaming ingestion (Auto Loader, Kafka, and Structured Streaming);

• Skills in dimensional modeling and Data Vault;

• Experience with dbt and/or Delta Live Tables (DLT);

• Understanding of data quality frameworks (Great Expectations, DLT Expectations);

• Capability in observability and pipeline monitoring;

• Experience in orchestration with Databricks Workflows and/or Apache Airflow;

• Integration expertise with APIs and diverse data sources;

• Proficiency in English is required for effective communication.

🏝️ Benefits

• Meal or food allowance (meal voucher);

• Discounts on courses, universities, and language schools;

• Stefanini Academy — a platform offering free, current online courses with certifications;

• Mentoring opportunities;

• Benefits club for medical consultations and examinations;

• Health insurance;

• Dental insurance;

• Perks and discounts at top establishments;

• Travel club;

• Pet care plan/partnership.