Senior Data Engineer – Azure Databricks

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, maintain, and enhance data pipelines utilizing Azure Databricks, PySpark, Azure Data Factory, Python, and Azure Data Lake.

• Construct and assist ETL/ELT processes for comprehensive data integration and analytical requirements.

• Execute data ingestion, transformation, validation, and reconciliation across various systems.

• Perform data analysis and profiling to identify quality issues, ensuring the reliability and consistency of information.

• Create, write, and refine complex SQL queries for optimal data processing performance.

• Conduct performance tuning and troubleshooting for large-scale workloads.

• Address Level 2 production incidents related to Azure Data Factory pipelines, Databricks jobs, and data processing workflows.

• Oversee and support production pipelines to guarantee availability, scalability, and operational efficiency.

• Facilitate deployment and release processes across development, staging, and production environments.

• Manage multiple project priorities and demands while working autonomously and maintaining a strong focus on meeting deadlines.

⛳️ Requirements

• Over 8 years of experience in Data Engineering.

• Demonstrated experience in Data Engineering and the development of ETL processes.

• Significant hands-on experience with Azure Databricks and Azure Data Factory (ADF).

• Advanced expertise in ETL/ELT development and data transformation.

• Proficient in SQL, including the development, optimization, and tuning of complex queries.

• Experience with Python and PySpark.

• Familiarity with Azure Data Lake and enterprise data integration.

• Background in data analysis, validation, profiling, and quality assurance.

• Experience in supporting, monitoring, and troubleshooting production pipelines.

• Capability to work independently, remain organized, and manage priorities effectively.

• Strong verbal and written communication skills are essential.

• Advanced proficiency in English is required.

• Preferred: experience with CI/CD pipelines and DevOps methodologies.

• Preferred: knowledge of Git, Azure DevOps, or similar version control and deployment tools.

• Preferred: familiarity with monitoring, logging, and observability tools in cloud environments.

🏝️ Benefits

• CLT employment contract.

• Fully remote work.