Engenheiro de Dados SR – Databricks

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Develop, enhance, and maintain scalable data pipelines for ingestion, transformation, and availability of information in a Databricks environment, utilizing Python, SQL, Spark, and DBT;

• Design, implement, and optimize ETL/ELT processes, ensuring high performance, reliability, governance, and data quality throughout the lifecycle of the solutions;

• Build, maintain, and monitor workflows and DAGs in Apache Airflow, ensuring proper orchestration, automation, and observability of data loads;

• Engage in modeling and implementing Bronze, Silver, and Gold layers following the Lakehouse architecture and best practices in Data Engineering;

• Integrate data from multiple sources, including APIs, relational and non-relational databases, enterprise systems, and cloud services;

• Support the definition of technical standards, best development practices, versioning, testing, and documentation of data pipelines;

• Collaborate with Analytics, BI, Data Science teams, and business areas to understand requirements and transform needs into scalable data solutions;

• Investigate and resolve incidents, performance issues, and failures in data processes, ensuring the stability and availability of the platform.

⛳️ Requirements

• Solid experience as a Senior Data Engineer in corporate data projects and high-volume information environments;

• Proficiency in Databricks, Apache Spark, Delta Lake, Python, and SQL for the development, maintenance, and optimization of data pipelines;

• Practical experience with DBT for data transformation, modeling, and asset management;

• Advanced knowledge of Apache Airflow for orchestration, monitoring, and automation of data flows;

• Experience with Lakehouse architecture and data modeling in Bronze, Silver, and Gold layers;

• Background in data integration through REST APIs, relational databases, and other corporate sources;

• Familiarity with code versioning using Git and experience with Azure DevOps or similar CI/CD tools;

• Experience with quality practices, monitoring, observability, and data governance;

• Knowledge in performance tuning and optimization of queries and data processing pipelines;

• Certifications in Databricks, Azure Data Engineer Associate, or similar qualifications;

• Experience with Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), and other Azure cloud data services;

• Understanding of DataOps, CI/CD for data pipelines, and Infrastructure as Code (IaC);

• Experience with Data Quality, Data Governance, and Data Observability tools.

🏝️ Benefits

• Position also available for individuals with disabilities (PcD).