Senior AI Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• Design and construct pipelines for data ingestion into Databricks from various sources, including SAP and web scraping.

• Develop scenario-based tests and examine edge cases.

• Engage with stakeholders to clarify requirements and finalize code modifications along with acceptance criteria.

• Set up Unity Catalog permissions, OBO tokens, and Service Principals for Databricks AI and machine learning applications.

• Create and deploy Databricks Apps using Visual Studio Code.

• Dispatch custom notifications from Databricks via APIs to specific Team Channel Webhooks.

• Develop minimal single-page chat applications utilizing FastAPI, React, and Claude Models/APIs.

• Implement system prompts, user prompts, prompt engineering, token economics, and prompt-based rule enforcement.

• Reengineer code and align its functionality with user requirements.

• Utilize graphs to determine rule interdependencies and identify circular references.

• Process text and images with the Claude Vision API.

• Create Retrieval Augmented Generation solutions utilizing Databricks AI Search indexes and hierarchical chunking.

• Establish fundamental AI safety guardrails.

• Employ caching mechanisms.

• Collaborate with Claude Code and Genie.

⛳️ Requirements

• Proven experience in designing and constructing data ingestion pipelines into Databricks from sources such as SAP and web scraping.

• Strong technical skills in PySpark and PySQL.

• Experience in developing scenario-based tests and assessing edge cases.

• Capability to work collaboratively with stakeholders to finalize code alterations and acceptance criteria.

• Familiarity with the Databricks Unity Catalog permissions model.

• Proficiency in using OBO tokens and configuring Service Principals for Databricks Agents, Genie Spaces, Machine Learning, and Foundation Models.

• Understanding of Databricks Apps.

• Ability to create and deploy Databricks Apps utilizing Visual Studio Code.

• Skill in sending custom notifications from Databricks through APIs to designated Team Channel Webhooks.

• Competence in employing FastAPI and React to develop minimal single-page applications with chat UI interfaces and Claude Models/APIs.

• Expertise in prompt engineering, including the use of system and user prompts.

• Basic understanding of token economics and rule enforcement within prompts.

• Capability to reengineer code, comprehend its functionality, and relate it to user requirements.

• Ability to analyze graphs to uncover interdependencies among rules and circular references.

• Proficient in utilizing the Claude Vision API along with standard text processing.

• Knowledge in the development of Retrieval Augmented Generation for Databricks, including AI Search indexes and hierarchical chunking.

• Awareness of essential AI safety guardrails.

• Understanding of caching strategies.

• Experience working with Claude Code and Genie.

🏝️ Benefits

• Competitive salary and performance-based incentives.

• Comprehensive health, dental, and vision insurance.

• Opportunities for professional development and continuous learning.

• Flexible working hours and remote work options.

• Collaborative and innovative work environment.