Senior Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design and execute a comprehensive medallion architecture within Databricks.

• Ingest data into Azure Data Lake Storage Gen2 and create Bronze, Silver, and Gold layers for both batch and near-real-time data sources.

• Facilitate data ingestion from source systems into Azure, managing schema evolution, validation, and reprocessing.

• Utilize Unity Catalog for effective access control, data lineage, and governance.

• Develop and sustain CI/CD pipelines within Azure DevOps.

• Orchestrate production pipelines leveraging Databricks Workflows.

• Collaborate in platform ownership with the Analytics Engineer, ensuring clear documentation of responsibilities and timelines.

• Oversee ingestion processes, transformation logic, and the integrity and structure of the Gold layer.

• Deliver a Gold layer that supports insights, business intelligence, and assets primed for Data Science.

⛳️ Requirements

• Proven experience in Data Engineering, with a minimum of 3 years of practical experience in Databricks and medallion architecture within a production environment.

• Strong proficiency in Python, SQL, and PySpark.

• Previous experience with Azure Data Lake Storage Gen2, Unity Catalog, Azure DevOps, and Databricks Workflows.

• Fluent English is essential, as this position involves direct collaboration with a global program team.

• Preferred: experience in integrating Salesforce as a data source.

• Preferred: Databricks certification, such as Databricks Certified Data Engineer Professional or Associate.

• Preferred: experience in the automotive sector or in other regulated, multi-site manufacturing environments.

• Demonstrated proactivity, effective communication, ownership, and a genuine commitment to the role.

🏝️ Benefits

• Competitive salary and performance-based incentives.

• Comprehensive health and wellness programs.

• Opportunities for professional development and growth.

• Flexible working hours and remote work options.

• Engaging and collaborative work environment.