Lead/Sr AI Engineer, Azure Data Bricks

This is a fully remote position, open to applicants in India.

📋 Description

• Develop production-ready AI applications and facilitate their deployment on Azure Databricks and the Azure cloud platform.

• Create comprehensive data and AI pipelines utilizing Azure Databricks.

• Design and implement ETL/ELT workflows using Python/PySpark and SQL.

• Establish CI/CD pipelines for Databricks jobs, notebooks, and workflows.

• Integrate Databricks with Azure Data Lake, Blob Storage, Key Vault, Azure OpenAI, Azure Functions, and other related services.

• Enhance job performance, cost efficiency, and reliability.

• Construct reusable and modular code.

• Work collaboratively with data scientists, platform engineers, and business stakeholders to transition models from experimentation to production.

• Set up logging, monitoring, and error handling for production pipelines.

• Create and deploy ML and Generative AI models, including LLMs, embeddings, and RAG pipelines, for applications in NLP, computer vision, and predictive analytics.

• Fine-tune LLMs using LoRA/QLoRA and integrate them with Azure OpenAI or Hugging Face models.

• Implement vector search and retrieval pipelines utilizing FAISS or Azure Cognitive Search.

• Adhere to responsible AI practices, including bias detection and model governance.

• Ingest data into Databricks from various sources such as SAP and web scraping.

• Develop scenario-based and edge-case tests, finalizing code changes and acceptance criteria with stakeholders.

• Dispatch custom notifications from Databricks through APIs and custom Teams channel webhooks.

• Create chat UI interfaces and Claude-based applications.

• Utilize graphs to identify rule dependencies and circular references.

• Re-engineer code to align with user requirements.

• Establish AI safety guardrails and caching mechanisms.

⛳️ Requirements

• Proficient experience with Azure Databricks (jobs, workflows, clusters, with Unity Catalog preferred).

• Strong skills in Python, particularly with PySpark over pandas.

• Proficiency in SQL, including complex joins, window functions, and analytical queries.

• Knowledge of Azure Cloud, including ADLS Gen2, ADF, Key Vault, and IAM concepts.

• Experience in pipeline orchestration and deployment, including CI/CD and environment promotion.

• Familiarity with Azure DevOps.

• Strong comprehension of the ML lifecycle and MLOps best practices.

• Experience in deploying models using MLflow or similar frameworks.

• Exposure to ML and Generative AI workloads on Databricks.

• Experience with RAG, embeddings, or inference pipelines.

• Knowledge of Terraform, ARM, or Bicep.

• Familiarity with Databricks Asset Bundles.

• Understanding of Airflow or ADF orchestration.

• Experience in production monitoring and cost optimization.

• Knowledge of LangChain or similar frameworks.

• Familiarity with Azure AI services, including Azure Machine Learning and Azure Cognitive Services.

• Understanding of Databricks Unity Catalog permissions, OBO tokens, and Service Principals.

• Ability to build and deploy Databricks Apps using Visual Studio Code.

• Capability to use FastAPI and React for developing minimal single-page applications.

• Skills in prompt engineering, including system and user prompts.

• Basic understanding of token economics and prompt rule enforcement.

• Ability to utilize the Claude Vision API and standard text processing techniques.

• Comprehension of Retrieval Augmented Generation, including AI Search indexes and hierarchical chunking.

• Knowledge of basic AI safety guardrails and caching mechanisms.

🏝️ Benefits

• Opportunity for professional growth and development.

• Competitive salary and performance-based incentives.

• Access to cutting-edge technology and tools.

• Collaborative and innovative work environment.

• Flexible work arrangements and work-life balance initiatives.