Senior Databricks Data Engineer

📋 Description

• Construct the new corporate Lakehouse utilizing Databricks on AWS;

• Design and implement batch and streaming data pipelines using PySpark and Spark SQL;

• Establish and oversee the Medallion architecture (Bronze, Silver, Gold) within Delta Lake;

• Engage in the design of the new Data Warehouse data model;

• Transition data, pipelines, and routines from the existing Databricks on Azure setup to AWS;

• Move jobs and integrations from IBM DataStage, Azure Data Factory, and Azure Synapse Analytics;

• Transfer Databricks workspaces between Azure and AWS environments;

• Set up and manage data governance through Unity Catalog and Odin;

• Execute monitoring, testing, and ensure data quality;

• Collaborate with DevOps on deployment automation and CI/CD;

• Manage version control for notebooks, pipelines, and code via Git and Databricks Repos.

⛳️ Requirements

• Proven experience as a Data Engineer;

• In-depth knowledge of Databricks and Apache Spark;

• Proficiency in Python development and advanced SQL;

• Familiarity with AWS environments (S3, Glue Catalog, Lake Formation);

• Experience with Lakehouse architecture and Delta Lake;

• Understanding of dimensional modeling and Data Warehouse concepts;

• Background in developing batch and streaming pipelines;

• Knowledge of Git and CI/CD methodologies;

• Experience in cloud migration and data modernization initiatives.

• Nice to have:

• Exposure to IBM DataStage;

• Familiarity with Azure Data Factory and Azure Synapse Analytics;

• Experience in data governance and data cataloging.

🏝️ Benefits

• Competitive salary and performance bonuses;

• Comprehensive health and wellness benefits;

• Opportunities for professional development and training;

• Flexible work hours and remote work options;

• Collaborative and inclusive company culture.