Senior Data Engineer, Databricks

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Transition structures and data within Azure Databricks, encompassing notebooks, clusters, and catalogs.

• Collaborate with Unity Catalog, focusing on schemas, tables, external locations, and data governance.

• Enhance ingestion and transfer workflows for substantial data volumes utilizing ADLS Gen2, AzCopy, Delta, and Parquet.

• Oversee notebook parameterization, including widgets, libraries, and sizing of clusters.

• Automate processes using Python, PySpark, and SQL.

• Assist in validation, user acceptance testing, and troubleshooting in partnership with technical teams and vendors.

• Maintain data integrity, performance, and governance in critical operational environments.

⛳️ Requirements

• Extensive experience with Azure Databricks and proficiency in PySpark/Python.

• Familiarity with Azure Data Lake Storage Gen2, ACLs, and AzCopy.

• Advanced skills in SQL and data engineering.

• Experience with Unity Catalog and principles of data governance.

• Analytical, self-driven professional with excellent technical communication abilities.

🏝️ Benefits

• Comprehensive benefits package.

• Opportunities for professional development and growth.

• Flexible work arrangements and a supportive team environment.