Databricks Data Engineer

📋 Description

• Create and manage batch and streaming data pipelines in Databricks utilizing PySpark and Spark SQL;

• Construct pipelines in accordance with the established Medallion architecture (Bronze, Silver, and Gold);

• Aid in transferring legacy pipelines and jobs from tools such as IBM DataStage, Azure Data Factory, and Azure Synapse Analytics to Databricks Workflows;

• Assist in migrating routines and notebooks from Databricks on Azure to AWS;

• Develop and version control notebooks and code using Databricks Repos and Git;

• Implement fundamental tests and data quality procedures within pipelines;

• Collaborate with the architecture team on dimensional modeling of the Data Warehouse (Star Schema);

• Work with the DevOps team to automate deployments and CI/CD processes;

• Engage in the implementation of data governance with Unity Catalog;

• Monitor and provide assistance for production data pipelines.

⛳️ Requirements

• Proven experience as a Data Engineer or Mid-Level Data Engineer;

• Proficient in Databricks and Apache Spark (PySpark and Spark SQL);

• Strong experience with Python and advanced SQL;

• Familiarity with cloud platforms, preferably AWS (S3, Glue Catalog);

• Understanding of Lakehouse architecture and Delta Lake;

• Experience working with batch and/or streaming data pipelines;

• Acquainted with Git and code versioning practices;

• Basic understanding of dimensional modeling (Star Schema / Data Warehouse);

• Fundamental knowledge of CI/CD processes as applied to data;

• **Differentials:**

• Familiarity with IBM DataStage (migration or legacy support);

• Knowledge of Azure Data Factory and Azure Synapse Analytics;

• Experience with Databricks on Azure;

• Understanding of data governance and catalog solutions (Unity Catalog or similar).

🏝️ Benefits

• Competitive salary and performance-based bonuses;

• Opportunities for professional growth and development;

• Health, dental, and vision insurance;

• Flexible work hours and remote work options;

• Collaborative and innovative work environment.