Databricks Engineer

This is a fully remote position, open to applicants in Canada.

📋 Description

• Develop, enhance, and sustain efficient data pipelines utilizing SQL, dbt, and Python.

• Establish data-quality verification processes and automated testing frameworks within dbt to identify anomalies.

• Oversee production workflows and handle alerts effectively.

• Diagnose pipeline failures from the root cause to a permanent resolution.

• Create efficient PySpark transformations for extensive datasets within a Lakehouse setting.

• Convert intricate data challenges into straightforward action plans for technical team members and business stakeholders.

• Maintain comprehensive production ownership of data platforms, ensuring high data integrity and strong system performance.

⛳️ Requirements

• Advanced SQL expertise and substantial hands-on experience in developing modular models, snapshots, and custom schema tests in dbt.

• Strong capability in crafting clean, maintainable Python scripts for data manipulation, automation, and API integrations.

• Adequate practical experience using PySpark for data transformation and basic performance optimization, including joins and the DataFrame API.

• Functional knowledge of Delta Lake or comparable Lakehouse technologies, encompassing ACID transactions and MERGE operations.

• Outstanding written and verbal communication abilities.

• Exceptional troubleshooting skills with a systematic approach to pinpointing root causes and confirming fixes.

• Experience managing pipelines from start to finish, including CI/CD deployment, monitoring, and managing schema drift.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Opportunities for professional development and continuous learning.

• Flexible working hours and remote work options.

• Collaborative and innovative work environment.