Senior Data Engineer

This is a fully remote position, open to applicants in Canada.

📋 Description

• Collaborate with business stakeholders and product owners to comprehend objectives, requirements, and success criteria for data products.

• Design and implement scalable, secure, and high-performance data architecture on Microsoft Azure within cloud-native and hybrid environments.

• Lead the development of data ingestion, transformation, and integration pipelines utilizing Azure Data Factory, Azure Databricks, and Azure Synapse Analytics.

• Oversee data lakes and structured storage solutions using Azure Data Lake Storage Gen2, ensuring efficient access and governance.

• Integrate data from a variety of source systems, such as ServiceNow and geospatial systems, employing APIs, connectors, and custom scripts.

• Develop and maintain data models and semantic layers for reporting, analytics, machine learning, and downstream usage.

• Construct and optimize Python and SQL data workflows for data cleansing, enrichment, and advanced analytics within Azure Databricks.

• Design and expose secure data services and APIs through Azure API Management.

• Implement metadata management, data classification, and lineage tracking.

• Ensure compliance with privacy and regulatory standards through role-based access controls, encryption, and data masking.

• Monitor and troubleshoot data pipelines and integrations to guarantee reliability, scalability, and performance.

• Leverage AI and automation tools for the development, testing, monitoring, and documentation of pipelines.

• Utilize AI-assisted tools for code generation, optimization, and review.

• Design and curate standardized, high-quality datasets for advanced analytics and future AI applications.

• Perform additional duties as required.

⛳️ Requirements

• Proven experience in designing scalable, secure, and high-performance data architecture on Microsoft Azure.

• Familiarity with cloud-native and hybrid environments.

• Demonstrated experience in developing data ingestion, transformation, and integration pipelines.

• Proficiency in using Azure Data Factory, Azure Databricks, and Azure Synapse Analytics.

• Experience managing Azure Data Lake Storage Gen2 and structured storage solutions.

• Background in integrating ServiceNow and geospatial systems utilizing APIs, connectors, and custom scripts.

• Expertise in developing data models and semantic layers for reporting, analytics, and machine learning.

• Strong proficiency in Python and SQL.

• Experience with Azure API Management and secure data services/APIs.

• Knowledge of metadata management, data classification, and lineage tracking.

• Familiarity with privacy and regulatory standards, including FOIP and GDPR.

• Understanding of role-based access controls, encryption, and data masking.

• Proven ability in monitoring and troubleshooting data pipelines and integrations.

• Experience utilizing AI and automation tools for data engineering processes.

• Familiarity with AI-assisted code generation, optimization, and review.

🏝️ Benefits

• Competitive salary and comprehensive benefits package.

• Opportunities for professional development and career advancement.

• Flexible working hours and potential remote work options.

• A collaborative and innovative work environment.