Lead Data Consultant – Microsoft Fabric

This is a fully remote position, open to applicants in Alabama, +18 more states.

📋 Description

• Assist in a client engagement aimed at modernizing an Azure Synapse data platform and migrating selected workloads to Microsoft Fabric.

• Assess and enhance the architecture of Microsoft Fabric, including metadata structures, event triggering, and reusable ingestion patterns.

• Create and deploy metadata-driven ingestion frameworks that allow for configuration-based onboarding.

• Transition data sources and workloads from Azure Synapse to Microsoft Fabric.

• Develop and manage Fabric pipelines and Spark notebooks utilizing PySpark and Spark SQL.

• Examine existing Synapse pipelines and convert processing logic into Fabric solutions.

• Establish validation and reconciliation processes to ensure migration accuracy, such as duplicate detection, key alignment, and record comparisons.

• Suggest loading patterns including full-load, incremental, or CDC options.

• Implement Git-based CI/CD and automated promotion of environments for Fabric artifacts.

• Utilize Claude Code to set specifications, accelerate development, and review AI-generated code.

• Assess requirements for data dictionaries, metadata, lineage, and catalogs.

• Work alongside client engineers through paired development, code reviews, and knowledge-sharing sessions.

• Apply industry best practices for performance, testing, monitoring, documentation, security, and operational support.

⛳️ Requirements

• Candidates must reside within the United States.

• W2 employees should live in Alabama, Colorado, Delaware, Florida, Georgia, Iowa, Kansas, Kentucky, Louisiana, Minnesota, Missouri, Montana, Nebraska, Nevada, North Carolina, Ohio, South Carolina, Tennessee, Texas, Virginia, or Wisconsin.

• Must be authorized to work in the United States without sponsorship now or in the future.

• Ability to pass a background check is required.

• At least 5 years of experience working with Azure Data Services, including Microsoft Fabric, Azure Data Factory, Databricks, or Synapse.

• Practical experience with Microsoft Fabric in production environments, including lakehouses, pipelines, notebooks, medallion architecture, workspaces, and capacity management.

• Strong background in building and maintaining production ETL/ELT pipelines and reusable data ingestion frameworks.

• Proven experience in designing and implementing metadata-driven ingestion frameworks.

• Skilled in developing Spark notebooks and data transformation workflows using PySpark and Spark SQL.

• Strong SQL proficiency for data transformation, validation, and performance optimization, including T-SQL.

• Familiarity with Azure Synapse and complex data processing dependencies.

• Experience in implementing Bronze, Silver, and Gold medallion architecture patterns.

• Experience in designing and executing Git-based CI/CD workflows using GitHub Actions, Azure DevOps, or similar tools.

• Practical experience using Claude Code for AI-assisted software development.

• Experience in validating data migrations and ensuring data quality, accuracy, and consistency.

• Familiarity with integrating data from REST APIs, relational databases, and structured or semi-structured data sources.

• Understanding of data governance, lineage, metadata management, and data catalog concepts.

• Ability to engage in client-facing technical discussions.

• Excellent written and verbal communication skills.

• Preferred: experience in migrating Azure Synapse workloads to Microsoft Fabric; knowledge of Oracle and DB2; expertise in incremental ingestion and CDC; experience with migration reconciliation frameworks; familiarity with Microsoft Purview; consulting or professional services experience.

🏝️ Benefits

• Opportunity to work remotely from anywhere in the United States with flexible work hours.

• Paid time off, company holidays, and floating holidays.

• Comprehensive benefits package with contributions toward premiums.

• 401(k) plan available.

• Sponsored and supported learning opportunities.

• Options for W2 or 1099 engagement.