Databricks Engineer

This is a fully remote position, open to applicants in Canada.

📋 Description

• Spearhead the deployment of contemporary data platforms and architectures on Databricks, which includes ETL processes, workload migrations, and Unity Catalog integration.

• Create and execute data pipelines utilizing Azure Databricks, focusing on PySpark and SQL.

• Develop and enhance both batch and streaming data workloads.

• Transition legacy data workloads to Azure and Databricks environments.

• Adopt Delta Lake methodologies, including medallion architecture, change data capture (CDC), and data quality assurance.

• Connect Databricks with Azure services such as ADLS, ADF, Azure Key Vault, and DevOps/GitHub.

• Improve performance and manage costs through cluster sizing, job orchestration, and query optimization.

• Collaborate effectively with solution architects, analytics engineers, and client stakeholders.

• Contribute to the development of reusable accelerators, standards, and internal best practices.

• Facilitate client enablement via knowledge transfer and comprehensive documentation.

• Deliver hands-on solutions, guiding client engineers while ensuring adherence to best practices.

• Establish governance frameworks and implement Unity Catalog, focusing on data access, lineage, and security protocols.

• Assess and prioritize impactful AI and ML use cases, incorporating Generative AI into client strategies.

• Serve as a thought leader by engaging in client workshops, executive roundtables, and industry dialogues.

⛳️ Requirements

• 5–7+ years of experience in the field of data engineering.

• 3+ years of practical experience with Databricks, including its advanced features such as Delta Lake, Unity Catalog, and MLflow.

• Demonstrated experience in leading extensive data migrations, including ETL processes, workloads, and cloud platforms.

• Strong proficiency in Azure environments.

• Multi-cloud experience is considered an advantage.

• Familiarity with Azure data services, including ADLS, ADF, Synapse, and others.

• Comprehensive understanding of modern data architectures such as lakehouse, medallion, and ELT/ETL frameworks.

• Experience with CI/CD processes for data workloads.

• Excellent communication skills with the ability to engage directly with clients.

• Strong understanding of data governance, security, and compliance within enterprise settings.

• Consulting experience is highly preferred.

🏝️ Benefits

• Engage in challenging, high-impact projects with mid-sized and enterprise clients across various industries.

• Collaborate with a senior, high-performing team that prioritizes speed, pragmatism, and client success.

• Databricks or cloud certifications are required.