Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, maintain, and enhance data processing systems and ETL pipelines utilizing tools such as Azure Data Factory and Azure Databricks.

• Partner with the Data Architect to establish data models and architectures that facilitate scalable and dependable data flows.

• Guarantee data quality and integrity through unit testing with Pytest and through component and integration testing with Behave.

• Collaborate with cross-functional teams to assimilate data from diverse sources by employing PySpark processes executed in Databricks and orchestrated by Azure Data Factory.

• Oversee infrastructure components within the Azure Cloud, incorporating role-based access control and Infrastructure as Code (IaC) methodologies.

• Participate in continuous integration and continuous deployment (CI/CD) processes utilizing Azure DevOps and related tools.

• Conduct data analysis and processing using SQL, Python, and Spark to meet business intelligence and reporting requirements.

⛳️ Requirements

• Over 5 years of experience in data engineering or a similar field.

• Expertise in SQL, Python, and Spark for data processing and analysis.

• Significant experience with Azure Data Factory and Azure Databricks in the development of ETL and data pipelines.

• Understanding of data modeling and architecture concepts.

• Familiarity with data quality tools and methodologies, including automated testing with PyTest and Behave.

• Basic knowledge of Azure Kubernetes Service (AKS) and event-driven architectures.

• Understanding of CI/CD methodologies and tools such as Azure DevOps, YAML, and shell scripting.

• Experience in infrastructure management within Azure, with a foundational knowledge of Terraform and role-based access control.

🏝️ Benefits

• Competitive salary and benefits package.

• Opportunity to join at the ground floor with a purpose-driven mindset.

• Development and career advancement paths to enhance your abilities.

• Attractive bonus potential from day one.