Senior Azure Data Engineer

This is a fully remote position, open to applicants in Illinois.

📋 Description

• Design and develop robust, reusable, and parameter-driven ingestion and transformation pipelines utilizing Azure Data Factory, Synapse Pipelines, Data Bricks, and/or Microsoft Fabric Data Factory.

• Implement medallion architecture (Bronze / Silver / Gold) on Azure Data Lake Storage Gen2 employing Delta Lake, Parquet, and structured streaming methodologies.

• Construct efficient ELT workflows that capitalize on pushdown to source systems (Synapse Dedicated SQL Pool, Azure SQL, Teradata) when applicable.

• Create and optimize PySpark notebooks and jobs on Azure Databricks or Synapse Spark.

• Design dimensional models (Kimball star/snowflake) and data vault patterns for analytics purposes.

• Implement Slowly Changing Dimensions (Type 1/2/3), Change Data Capture, and patterns for late-arriving data.

• Optimize distributed SQL workloads in Synapse Dedicated SQL Pool / Fabric Warehouse, focusing on distribution keys, partitioning, and clustered column store indexes.

• Establish CI/CD for data pipelines using Azure DevOps (YAML pipelines, ARM/Bicep/Terraform) across Development, SIT, UAT, and Production environments.

• Equip pipelines with comprehensive logging, auditing, and monitoring through Azure Monitor, Log Analytics, and KQL.

• Establish and maintain coding standards, code review processes, branching strategies, and release management protocols.

• Lead or assist in migrations from legacy systems to cloud platforms — for instance, transitioning from Informatica PowerCenter to Azure Data Factory, or moving on-premises Teradata / Oracle / SQL Server to Synapse or Fabric.

• Conduct workload assessments, capacity planning, and cost modeling for target-state architectures.

• Respond to production incidents for critical pipelines.

⛳️ Requirements

• Extensive hands-on experience with Azure Data Factory: pipelines, datasets, linked services, triggers, parameterization, mapping data flows, and all three Integration Runtime types (Azure, Self-hosted, SSIS).

• Strong experience in Data Bricks and PySpark.

• Production experience with one or more of the following: Azure Synapse Analytics (Dedicated and Serverless SQL Pools, Spark Pools), Azure Databricks (Delta Lake, Unity Catalog), or Microsoft Fabric (Warehouse, Lakehouse, OneLake).

• Solid understanding of Azure Data Lake Storage Gen2 (hierarchical namespace, RBAC + ACLs, lifecycle management, security).

• Familiarity with Azure Key Vault, Azure AD / Entra ID (including managed identities and service principals), and private networking (VNet integration, private endpoints).

• Experience in monitoring and troubleshooting using Azure Monitor, Log Analytics, and KQL.

• Advanced SQL skills — including window functions, CTEs, query optimization, execution plan analysis, and performance tuning.

• Proficient in Python for data engineering — including pandas, PySpark, REST API integration, and unit testing (pytest).

• Competent in T-SQL; knowledge of Spark SQL, KQL, PowerShell, and Bash shell scripting is a plus.

• Over 5 years of experience in data warehouse development.

• More than 5 years of experience in data modeling using ERWIN or similar tools.

• At least 2 years of experience with Azure Data Factory and Snowflake.

• Familiarity with the Medicaid domain is advantageous.

🏝️ Benefits

• Health insurance

• 401(k) matching

• Flexible work hours

• Opportunities for professional development