AI/ML Engineer, Databricks

This is a fully remote position, open to applicants in India.

📋 Description

• Develop and implement feature pipelines as well as training and serving pipelines for machine learning and Generative AI applications utilizing MLflow.

• Facilitate the deployment of Generative AI and agentic systems on the Databricks platform, including preparing data for RAG pipelines, integrating vector search, and utilizing Databricks Genie Spaces.

• Establish new AI use cases from inception to completion on Databricks, encompassing data preparation and the creation of functional prototypes.

• Collaborate with data scientists to transition concepts from notebooks into production environments.

• Create reusable templates and accelerators within Databricks for RAG, agentic assistants, forecasting, and personalization applications.

• Enhance Databricks jobs, clusters, and workflows to optimize cost, performance, and reliability at scale in production.

• Engage directly with client engineering teams and stakeholders to collect requirements and resolve issues.

• Provide documentation and runbooks for seamless handover.

⛳️ Requirements

• Over 3 years of experience as a Machine Learning Engineer or in a related position.

• Significant hands-on experience with the Databricks platform.

• Proficient in Python and PySpark programming.

• Capability to write high-quality production Spark jobs beyond just notebooks.

• Practical experience in delivering production-level Generative AI or agentic systems.

• Familiarity with RAG architectures, LLM orchestration frameworks like LangChain or LangGraph, and AgentOps tools.

• Extensive hands-on experience in the industry with data science and machine learning using pandas, scikit-learn, and PyTorch.

• Background in supporting large-scale consumer or enterprise ML systems, such as recommenders, ranking, forecasting, or personalization.

• Experience with MLflow or similar tools for experiment tracking, model registry, and model serving.

• Strong SQL skills.

• Familiarity with cloud data platforms like Azure, AWS, or GCP.

• Comfortable collaborating directly with clients or cross-functional stakeholders to define and execute technical projects.

• Preferred practical experience with Delta Lake and lakehouse/medallion architecture patterns.

• Working knowledge of Unity Catalog or similar data governance and access-control tools is preferred.

• Previous consulting or professional services experience in AI/ML delivery is preferred.

• Familiarity with experimentation and A/B testing pipelines built on Databricks is preferred.

• Exposure to causal inference or applied statistics is preferred.

• Databricks Certified Machine Learning Associate or Professional certification is preferred.

• Databricks Certified Generative AI Engineer Associate certification is preferred.

🏝️ Benefits

• Government-mandated benefits along with supplemental health insurance.

• A collaborative and professional work environment.

• Opportunities for career growth within a rapidly expanding organization.

• Flexible remote work arrangement.

• Training programs to support employee growth and career advancement.