Senior Data Engineer – Data & AI

This is a fully remote position, open to applicants in Texas.

📋 Description

• Oversee the design and development of comprehensive data platforms that encompass ingestion, cleaning, preparation, transformation, analytics, and activation.

• Develop Medallion (Bronze, Silver, Gold) lakehouse structures utilizing OneLake/Delta Lake.

• Take charge of data modeling for client projects across dimensional, normalized, and semantic layers.

• Create batch and streaming ingestion frameworks using Fabric Data Factory/pipelines, Azure Databricks Structured Streaming, and event-driven architectures such as Event Hubs.

• Identify and address architectural risks and technical debt; enhance pipeline performance and cost efficiency.

• Develop APIs and data services for regulated data activation.

• Design MCP and API-based integrations to link business systems, Data Agents, and agentic AI interfaces.

• Collaborate with AI and Architecture teams to make enterprise data accessible to Data Agents and Generative AI applications.

• Establish data governance frameworks using Microsoft Purview and/or Unity Catalog.

• Provide support for production operations and incident response related to critical pipelines and activation services.

• Guide Data Engineers and offer technical direction through code, model, and design reviews.

• Counsel client stakeholders on architectural trade-offs and design choices.

• Contribute to the development of reusable frameworks, accelerators, and engineering standards.

• Assist in presales activities, including estimates, proposals, and solution designs.

• Assess new Fabric and Azure Databricks features for client platforms.

⛳️ Requirements

• Bachelor's or Master's degree in Computer Science, Engineering, Mathematics, Statistics, or a related technical field.

• 5-8 years of experience in data engineering, with extensive ownership of production data pipelines on Azure.

• Expert-level skills in Python, SQL, and PySpark for constructing distributed data pipelines at scale.

• In-depth hands-on experience with the Microsoft Azure Data & AI ecosystem (Azure Data Factory, Microsoft Fabric, Synapse, ADLS Gen2) and/or Azure Databricks, including Delta Lake/OneLake and Medallion architectural patterns.

• Strong foundation in data modeling, schema design, and enterprise data architecture throughout the entire lifecycle from ingestion to activation.

• Experience in designing APIs and MCP-based integrations that facilitate data exchange with front-line systems, Data Agents, and other agentic AI interfaces.

• Familiarity with orchestration and CI/CD (Fabric pipelines, Azure Data Factory, Databricks Workflows, Azure DevOps).

• Solid understanding of data governance and security tools, including Microsoft Purview and/or Unity Catalog.

• Proven ability to mentor engineers and spearhead technical design discussions during client-facing projects.

• Strong problem-solving capabilities and the ability to independently troubleshoot complex, large-scale data challenges.

• Excellent communication skills, with the capability to articulate technical trade-offs to both technical and client audiences.

• Relevant Microsoft certifications (e.g., Azure Data Engineer Associate, Fabric Analytics Engineer Associate) and/or Databricks certifications are preferred.

🏝️ Benefits

• Opportunity to work on cutting-edge data technologies.

• Collaborative and innovative work environment.

• Professional development opportunities and career growth.

• Competitive salary and comprehensive benefits package.