Data Engineer – AI

This is a fully remote position, open to applicants in India.

📋 Description

• Assist in enterprise AI and Generative AI projects by creating scalable, secure, and AI-compatible data platforms.

• Collaborate with Data Engineers, AI Engineers, Data Scientists, and business partners to construct high-quality data pipelines.

• Facilitate Retrieval-Augmented Generation (RAG) solutions and ready enterprise data for AI-driven applications.

• Design, develop, and sustain scalable data pipelines utilizing Azure Databricks, Snowflake, and Azure Data Factory.

• Create ELT/ETL pipelines to gather data from ERP, CRM, manufacturing, supply chain, and various enterprise applications.

• Generate AI-ready datasets that cater to machine learning and Generative AI use cases.

• Consolidate structured, semi-structured, and unstructured data into centralized data platforms.

• Work alongside AI Engineers to securely expose enterprise data to LLM-powered applications.

• Enhance data models, transformations, and query performance tailored for analytics and AI tasks.

• Apply data quality, governance, lineage, monitoring, and security best practices.

• Develop reusable data transformation frameworks utilizing Python, SQL, Spark, and dbt (if applicable).

• Create REST API integrations to facilitate AI services and enterprise applications.

• Engage in architecture discussions, code reviews, CI/CD implementations, and Agile ceremonies.

• Assist with production deployments, monitoring, troubleshooting, and performance optimization.

⛳️ Requirements

• Over 5 years of experience in Data Engineering.

• Extensive hands-on experience with Azure Databricks and Snowflake.

• Proficient in Python, SQL, and PySpark.

• Experience in building scalable ELT/ETL pipelines using Azure Data Factory.

• Strong grasp of data warehousing, dimensional modeling, and data lake architectures.

• Familiarity with REST APIs and integration of cloud-based services.

• Knowledge of Git, Azure DevOps, and CI/CD pipelines.

• Experience in Agile/Scrum methodologies.

• Required experience in AI & Azure AI.

• Background in supporting enterprise AI or Generative AI initiatives.

• Working knowledge of Azure Machine Learning (Azure ML).

• Experience with Microsoft AI Foundry (Azure AI Foundry) for AI solution development and orchestration.

• Experience integrating Azure OpenAI Service into enterprise applications.

• Familiarity with Azure AI Search for enterprise search and Retrieval-Augmented Generation (RAG) solutions.

• Exposure to Microsoft Copilot Studio for creating AI-powered copilots and conversational experiences.

• Understanding of vector search, embeddings, prompt engineering, and LLM integration.

• Experience in preparing enterprise data for AI model training, inference, and knowledge retrieval.

🏝️ Benefits

• Flexible work arrangements.

• Opportunities for professional development.

• Options for remote work.