Senior Data Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, develop, and maintain robust production-grade data pipelines for batch, incremental, real-time, and near-real-time processing utilizing Microsoft Fabric and Azure.

• Execute Change Data Capture (CDC), watermarking, merge/upsert strategies, handling late-arriving data, and idempotent reprocessing methodologies.

• Create transformations using PySpark, Spark SQL, and T-SQL across bronze, silver, and gold layers in Delta Lake.

• Construct Fabric solutions leveraging OneLake, Lakehouse, Warehouse, Data Factory, Dataflows Gen2, Spark notebooks, Real-Time Intelligence, and Power BI.

• Implement dimensional models and Data Vault architectures, ensuring data quality testing, reconciliation, observability, logging, alerting, and SLA monitoring.

• Establish CI/CD practices based on Git, with automated deployment through Azure DevOps or GitHub, utilizing Fabric deployment pipelines, REST APIs, Bicep, and Terraform.

• Apply Purview catalog for data lineage, enforce least-privilege access, manage sensitivity controls, encryption, and ensure secure connectivity.

• Assist in migrating data from SQL Server, SSIS, SSAS, legacy warehouses, and Synapse to Fabric/Azure, encompassing conversion, reconciliation, and cutover processes.

• Prepare governed data for AI/ML and RAG workloads, focusing on ingestion, chunking, embeddings, indexing, and telemetry.

• Lead engineering workstreams, providing estimates, managing risks, communicating with clients, mentoring team members, developing reusable accelerators, and overseeing operational handover.

⛳️ Requirements

• Over 8 years of experience in data engineering and data warehousing.

• More than 3 years of experience in delivering Azure solutions in production environments.

• Proficiency with Microsoft Fabric.

• Advanced skills in T-SQL, Spark SQL, and Python/PySpark; familiar with KQL and DAX.

• Extensive experience with Delta Lake, Spark optimization, medallion/lakehouse methodologies, CDC, and incremental loading.

• Strong expertise in dimensional modeling and a solid understanding of Data Vault 2.0.

• Hands-on experience with CI/CD practices, Git, automated testing, infrastructure as code, DataOps, and governance/security measures.

• Experience in professional services and client-facing roles.

• Authorization to work in the US with the ability to travel up to 15%.

• Preferred qualifications include DP-700, DP-600, DP-203 certifications; experience with streaming/event-driven architectures, RAG/GenAI, Fabric IQ/data agents, and industry-specific knowledge.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Retirement savings plan with company match.

• Opportunities for professional development and certifications.

• Flexible work hours and remote work options.