Senior Data Engineer – AI, LLM Solutions

This is a fully remote position, open to applicants in Pakistan, +3 more countries.

📋 Description

• Design and create enterprise-level data solutions utilizing Microsoft Fabric, Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, and Azure Databricks.

• Construct and refine scalable ETL/ELT pipelines for structured, semi-structured, and unstructured data.

• Develop solutions for Lakehouse, Data Warehousing, and Medallion Architecture.

• Design and execute frameworks for both batch and real-time data processing.

• Spearhead data modernization, migration, and integration projects from legacy systems.

• Establish data foundations for AI, Machine Learning, and Generative AI applications.

• Prepare, transform, and enhance datasets for AI applications, model training, and optimization.

• Design and implement RAG architectures and vector search technologies.

• Support implementations of Azure OpenAI, Microsoft Copilot, and enterprise LLM solutions.

• Collaborate with Data Scientists and AI Engineers to operationalize AI and machine learning initiatives.

• Create distributed data processing solutions using PySpark and Spark SQL.

• Develop automated integrations that connect APIs, SaaS applications, databases, and cloud services.

• Create reusable data engineering frameworks, components, and standards.

• Enhance workloads for performance, scalability, reliability, and cost-effectiveness.

• Implement data quality, validation, monitoring, and error-handling protocols.

• Establish data governance and cataloging practices using Microsoft Purview.

• Create data lineage, classification, cataloging, and access control measures.

• Ensure compliance with security, privacy, and regulatory standards such as SOC 2, NIST, and CMMC.

• Design and support semantic data models for Power BI and enterprise analytics.

• Facilitate self-service analytics and support KPI frameworks, executive dashboards, reporting, and predictive analytics.

• Provide technical guidance and mentoring to junior and mid-level data engineers.

• Engage in solution architecture, technical design evaluations, implementation planning, and presales efforts.

• Collaborate with clients and stakeholders to translate business requirements into scalable technical solutions.

• Generate technical documentation, engineering standards, and reusable implementation patterns.

• Diagnose issues in data pipelines, platforms, and integrations.

• Monitor and optimize data workloads and cloud resources.

• Assist with enhancements, upgrades, and ongoing maintenance of data platforms.

• Identify opportunities for improvements in data quality, automation, performance, and operational efficiency.

• Stay updated with advancements in Microsoft Fabric, Azure, AI, Generative AI, and data engineering technologies.

⛳️ Requirements

• Bachelor’s degree in Computer Science, Software Engineering, Information Technology, Data Science, or a related field.

• 8+ years of professional experience in Data Engineering, Data Platforms, or Cloud Data Solutions.

• 5+ years of hands-on experience with Microsoft Azure Data Services.

• Demonstrated experience delivering enterprise-scale data, analytics, and AI solutions.

• Strong background in modern cloud data platforms and data architectures.

• Experience in consulting, professional services, or enterprise environments.

• Excellent communication, stakeholder management, and problem-solving abilities.

• Proficiency in Microsoft Fabric, Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage Gen2, Azure Databricks, and Azure Event Hubs.

• Familiarity with Data Warehouse, Lakehouse, and Medallion Architecture.

• Proficiency in Python, SQL, PySpark, Spark SQL, Kusto Query Language, REST API integrations, and ETL/ELT development.

• Experience with SQL Server, PostgreSQL, MySQL, NoSQL databases, data modeling, and optimization.

• Knowledge of Azure OpenAI, Generative AI, Prompt Engineering, Retrieval-Augmented Generation, Vector Databases/Vector Search, Machine Learning Pipelines, NLP fundamentals, LLM integration strategies, and AI-ready data architecture.

• Experience with Power BI, Tableau, Semantic Models, Executive Reporting, and KPI Frameworks.

• Understanding of Microsoft Purview, Data Governance, Data Classification, Data Lineage, Data Cataloging, Access Control, Enterprise Security Standards, and SOC 2, NIST, and CMMC.

• Familiarity with Azure DevOps, Git, CI/CD Pipelines, Infrastructure as Code concepts, Containerization principles, and cloud resource optimization.

• Ability to design and implement scalable data solutions and comprehend enterprise data architecture.

• Capability to work independently in a remote and cross-functional environment.

• Ability to mentor engineers and provide technical leadership.

• Strong emphasis on security, quality, performance, and reliability.

• Ability to convert business requirements into technical solutions.

• Preferred certifications include Microsoft Certified: Fabric Analytics Engineer Associate, Microsoft Certified: Azure Data Engineer Associate, Microsoft Certified: Azure Data Scientist Associate, Microsoft Certified: Azure Solutions Architect Expert, Databricks Certified Data Engineer, and relevant AI, Machine Learning, or Generative AI certifications.

🏝️ Benefits

• Remote-first work environment.

• Chance to work on enterprise-scale Microsoft Fabric and Azure data platforms.

• Exposure to AI, Generative AI, Azure OpenAI, and LLM projects.

• Opportunity to engage with modern cloud and data technologies.

• Collaborative, innovation-oriented culture.

• Experience with a variety of enterprise and consulting projects.