Data Quality Engineer, Intermediate

This is a fully remote position, open to applicants in Canada.

📋 Description

• Validate the ingestion, transformation, and transfer of data throughout the Bronze, Silver, and Gold data layers.

• Assess the completeness, accuracy, conformity, lineage, business rules, and exception handling of data.

• Conduct testing for ETL/ELT processes, data warehouses, data lakes, and lakehouse solutions.

• Confirm source-to-target mappings, schema modifications, and transformation logic.

• Validate batch processing, incremental updates, and CDC (Change Data Capture) functionalities.

• Execute large-scale data validation and reconciliation tasks.

• Evaluate data quality, auditability, and comprehensive data lineage.

• Investigate and resolve data defects utilizing SQL and other analytical tools.

• Work collaboratively with engineering and business teams to ensure that data solutions adhere to functional and quality standards.

• Assist in test planning, defect tracking, and reporting activities using Azure DevOps or comparable tools.

⛳️ Requirements

• Must possess a current Secret level security clearance at the time of application.

• Proven experience in testing data pipelines, ETL/ELT processes, data warehouses, data lakes, or lakehouse solutions.

• Strong understanding of data validation methodologies, data quality controls, and source-to-target mapping.

• Experience in validating both structured and semi-structured data.

• Advanced SQL proficiency for data analysis, reconciliation, and defect resolution.

• Familiarity with Python and PySpark for data testing and validation purposes.

• Experience with Azure Databricks, Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), Delta Lake, Kafka and/or Event Hub, and Azure DevOps.

• Knowledge of data profiling and data quality assessment tools.

• Strong analytical, problem-solving, and troubleshooting abilities.

• Exceptional attention to detail and capability to excel in a fast-paced, data-centric environment.

🏝️ Benefits

• Work in a collaborative environment that encourages knowledge-sharing and mentorship.

• Commitment to inclusive hiring practices.

• Support for accommodations throughout the recruitment process.

• Opportunity to engage in complex, real-world client projects.