Databricks Engineer

This is a fully remote position, open to applicants in Canada.

📋 Description

• Spearhead the execution of cutting-edge data platforms and architectures on Databricks, encompassing ETL, workload migrations, and Unity Catalog.

• Architect and establish data pipelines utilizing Azure Databricks (PySpark, SQL).

• Develop and enhance batch and streaming data workloads.

• Transition legacy data workloads to Azure and Databricks.

• Apply Delta Lake patterns, including medallion architecture, CDC, and data quality measures.

• Integrate Databricks with Azure services such as ADLS, ADF, Azure Key Vault, and DevOps/GitHub.

• Improve performance and cost efficiency through cluster sizing, job orchestration, and query tuning.

• Collaborate with solution architects, analytics engineers, and client stakeholders.

• Contribute to reusable accelerators, establish standards, and promote internal best practices.

• Facilitate client enablement via knowledge transfer and comprehensive documentation.

• Deliver hands-on solutions, guiding and collaborating closely with client engineers to ensure best practices are followed.

• Establish governance models and Unity Catalog, focusing on data access, lineage, and security frameworks.

• Assess and prioritize high-impact AI and ML use cases, integrating Generative AI into client strategies.

⛳️ Requirements

• 5–7+ years of experience in the field of data engineering.

• Over 3 years of practical experience with Databricks, including its advanced features (Delta Lake, Unity Catalog, MLflow).

• Demonstrated experience in leading large-scale data migrations, covering ETL, workloads, and cloud platforms.

• Strong proficiency in Azure environments.

• Experience with multi-cloud environments is a valuable asset.

• Familiarity with Azure data services (ADLS, ADF, Synapse, etc.) is essential.

• Comprehensive understanding of modern data architectures, including lakehouse, medallion, and ELT/ETL methodologies.

• Experience with CI/CD practices for data workloads is required.

• Databricks or cloud certifications are mandatory.

🏝️ Benefits

• Opportunities for professional development.