Data Engineer – Silver Conversions, Scheduling

This is a fully remote position, open to applicants in Canada.

📋 Description

• Create conversion and transformation pipelines for the Silver layer utilizing Microsoft Fabric, Azure Data Factory, Spark, and Delta Lake.

• Develop and oversee the scheduling and orchestration of Silver pipelines, managing dependencies, retries, and failure handling.

• Leverage reusable PySpark libraries and metadata-driven configurations to enhance delivery speed.

• Implement standard error handling, logging, monitoring, and unit testing protocols.

• Facilitate reconciliation, checkpointing, replay, and recovery processes to ensure data integrity.

• Optimize Spark workloads for both performance and cost-effectiveness.

⛳️ Requirements

• At least 5 years of experience with Microsoft Fabric (Data Factory, Lakehouse), PySpark, Spark SQL, Delta Lake, Azure Data Factory, and Azure Storage.

• Proficient in pipeline orchestration and scheduling using Fabric Data Factory / Azure Data Factory.

• Demonstrated experience in data transformation using PySpark and Spark SQL.

• Familiarity with Microsoft Purview for cataloging and lineage tracking.

• Experience with Azure DevOps, Git, and CI/CD pipelines.

• Current Reliability security clearance.

• Must be eligible for Secret clearance.

🏝️ Benefits

• Collaborative work environment.

• Access to mentorship and coaching.

• Opportunities for knowledge sharing.

• Commitment to inclusive hiring practices.

• Accommodation support available throughout the recruitment process.