Senior Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, develop, and enhance scalable data pipelines utilizing services from the Azure ecosystem and Databricks.

• Create robust data ingestion, transformation, and provisioning processes (ETL/ELT), including scenarios for real-time ingestion.

• Implement and optimize solutions leveraging Azure Data Factory, Azure Data Lake Storage (ADLS Gen2), Databricks Jobs/Workflows, Azure Functions, Azure Synapse Analytics, and Event Hubs/Event Grid.

• Design, implement, and maintain Data Lakes and Lakehouse architectures (medallion: Bronze/Silver/Gold) on Azure Databricks.

• Develop solutions using Python, SQL, and PySpark, focusing on performance, scalability, and data quality.

• Handle large volumes of both structured and unstructured data.

• Ensure data quality, integrity, security, and governance throughout the entire pipeline, utilizing Unity Catalog and ensuring compliance with LGPD.

• Optimize queries, data processing, and data consumption to minimize costs and enhance performance.

• Collaborate with Analytics, Data Science, Software Engineering, and Architecture teams to create strategic solutions.

• Implement monitoring, observability, and failure management in data pipelines.

• Engage in defining standards, best practices, and the advancement of the platform's data architecture.

• Assist in the analysis and resolution of critical incidents related to the data environment.

⛳️ Requirements

• Demonstrated experience as a Data Engineer within Azure environments.

• Strong expertise in core Azure data services such as Data Factory, ADLS Gen2, Databricks, Synapse Analytics, and Azure Functions.

• Extensive experience in developing ETL/ELT pipelines.

• Advanced proficiency in Python and SQL.

• Experience in data modeling for analytical environments and Data Lakes/Lakehouse.

• Familiarity with Apache Spark or PySpark.

• Experience with version control systems using Git and CI/CD methodologies (preferably Azure DevOps).

• Knowledge of partitioning, query optimization, and distributed processing.

• Experience in monitoring, observability, and troubleshooting of data pipelines.

• Understanding of cloud data architecture and best practices for security, governance, and compliance with LGPD.

• Strong analytical abilities, problem-solving skills, and capacity to work in collaborative settings.

• Nice to have: Experience with Databricks Unity Catalog for data governance and cataloging.

• Nice to have: Familiarity with Delta Lake and medallion architecture (Bronze/Silver/Gold).

• Nice to have: Experience with Azure Data Factory and/or Databricks Workflows / Databricks Asset Bundles (DABs) for orchestration.

• Nice to have: Experience with event-driven architectures (Event Hubs, Event Grid).

• Nice to have: Experience in integrating legacy/ERP systems (e.g., SAP) into data pipelines.

• Nice to have: Knowledge of data quality frameworks (e.g., DQX - Databricks Labs, dbt, Great Expectations).

• Nice to have: Experience with real-time data ingestion (e.g., Event Hubs, Kafka).

• Nice to have: Experience with Docker and Kubernetes (AKS).

• Nice to have: Knowledge of infrastructure-as-code (Terraform or Bicep/ARM Templates).

• Nice to have: Experience in Machine Learning, AI, and MLOps projects.

• Nice to have: Experience in large enterprise environments and mission-critical projects.

• Nice to have: Azure and/or Databricks certifications (e.g., DP-203, Databricks Certified Data Engineer).

🏝️ Benefits

• The contract type can be PJ (contractor, without benefits) or CLT (employee).