Senior Databricks Engineer

This is a fully remote position, open to applicants in Canada.

📋 Description

• Design, construct, and sustain scalable ETL/ELT pipelines on the Databricks Lakehouse Platform utilizing PySpark, Spark SQL, and Delta Lake.

• Create and execute medallion (bronze/silver/gold) data architectures while ensuring data quality, governance, and lineage standards are upheld.

• Enhance Spark jobs and cluster settings for optimal performance and cost efficiency, which includes strategies for partitioning, caching, and autoscaling.

• Implement and oversee Unity Catalog for access control, data governance, and sharing of assets across workspaces.

• Develop and manage workflows through Databricks Workflows, Delta Live Tables, and CI/CD pipelines.

• Collaborate with data scientists, analysts, and business stakeholders to convert requirements into dependable data products.

• Set up engineering best practices, perform code reviews, and provide mentorship to junior data engineers.

• Monitor production pipelines, resolve failures, and lead root-cause analysis along with continuous improvement initiatives.

⛳️ Requirements

• Over 5 years of data engineering experience, with at least 3 years developing production solutions on Databricks and Apache Spark.

• Mastery in Python (PySpark) and advanced SQL skills.

• Extensive hands-on experience with Delta Lake, Unity Catalog, and the medallion architecture pattern.

• Significant experience with at least one major cloud provider (AWS, Azure, or GCP) and its essential data services.

• Demonstrated success in optimizing Spark performance and managing cluster costs.

• Familiarity with data modeling, warehousing principles, and the creation of dimensional/analytics-ready datasets.

• Expertise in Git-based version control, CI/CD, and infrastructure-as-code practices.

• A Bachelor's degree in Computer Science, Engineering, or an equivalent practical background.

🏝️ Benefits

• Accommodation provided during the hiring process.

• Willingness to sponsor employment visas.