Azure Data Engineer – Fabric

This is a fully remote position, open to applicants in Canada.

📋 Description

• Design and sustain ingestion and orchestration pipelines utilizing Fabric Data Factory and Azure Data Factory.

• Integrate various databases, APIs, files, and enterprise applications.

• Create ETL/ELT transformations using SQL, Python, PySpark, Fabric notebooks, and Dataflow Gen2.

• Implement OneLake, Lakehouse, Warehouse, and Delta Lake solutions employing medallion architecture when suitable.

• Develop full and incremental load patterns, change data capture, parameterized pipelines, and reusable ingestion frameworks.

• Construct dimensional models and curated datasets in collaboration with analysts and business stakeholders.

• Apply business rules, historical tracking, and source-to-target mappings.

• Establish data quality checks, automated tests, reconciliation, and validation processes.

• Manage schema changes, duplicate records, missing values, and late-arriving data.

• Enhance pipeline reliability through logging, monitoring, alerts, retries, dependency management, and safe reruns.

• Diagnose failures and document operational runbooks.

• Utilize Git and CI/CD to facilitate changes across environments.

• Optimize performance of SQL, Spark, and pipelines.

• Enforce agreed-upon access controls.

• Engage in code reviews and provide mentorship.

• Collaborate closely with architects, analysts, and clients from source analysis and development to testing, deployment, and production support.

⛳️ Requirements

• Extensive experience in building and managing production data pipelines, overseeing complex engineering tasks from deployment to support.

• Practical experience with Microsoft Fabric, including OneLake, Lakehouse or Warehouse, Fabric Data Factory pipelines, and notebooks.

• Proficient in SQL, Python, and PySpark.

• Hands-on experience with data processing in Apache Spark and Delta Lake.

• Azure experience with Azure Data Factory, ADLS Gen2, and Azure SQL.

• Experience in integrating Azure services with Fabric.

• Knowledge of ETL/ELT, orchestration, incremental processing, data modeling, and both structured and semi-structured data.

• Familiarity with Git, deployment automation, testing, data quality, monitoring, and performance tuning in production settings.

• Ability to engage directly with clients to clarify requirements, resolve issues, and manage dependencies.

• Capability to work autonomously and collaborate across delivery teams.

• Experience with Dataflow Gen2 and Power Query is valued.

• Familiarity with OneLake shortcuts or mirroring is valued.

• Experience in integrating Power BI semantic models is valued.

• Familiarity with Azure Databricks, Azure Synapse, Microsoft Purview, or ERP and SaaS integration is valued.

• Experience with streaming workloads, Fabric Real-Time Intelligence, KQL, or Eventstreams is valued.

• Holding a Microsoft Certified Fabric Data Engineer Associate certification or other relevant Azure or Microsoft data certifications is valued.

🏝️ Benefits

• Opportunities to take ownership and expand expertise.

• Learn alongside experienced professionals.

• Opportunities to contribute to the company's delivery methodologies.

• Opportunities to influence the development of a growing consulting practice.

• Engage in direct client work on significant challenges.