Lead Data Engineer, CICD, AI

This is a fully remote position, open to applicants in India.

📋 Description

• Modify source code to accommodate architectural requirements, including authentication modifications and Databricks X-forwarded headers while utilizing Apps.

• Engage in technical evaluations for setting up Azure resources for application deployment.

• Create technical and network architecture documentation for reviews and approvals by the cloud and security teams.

• Design and develop Databricks data ingestion pipelines from various sources, such as SAP and web scraping, using PySpark and PySQL.

• Conduct scenario-based and edge-case testing; work collaboratively with stakeholders to finalize code modifications and acceptance criteria.

• Configure and manage Databricks Unity Catalog permissions, OBO tokens, and service principals for Databricks Agents, Genie Spaces, Machine Learning, and Foundation Models.

• Build and deploy Databricks Apps utilizing Visual Studio Code.

• Send tailored Databricks notifications through APIs to custom Teams channel webhooks.

• Develop minimal SPA chat interfaces leveraging FastAPI, React, and Claude models/APIs.

• Implement system prompts, user prompts, prompt engineering, token economics, rule enforcement, Claude Vision API, NLP, RAG, AI safety guardrails, and caching strategies.

• Collaborate with Claude Code and Genie.

• Troubleshoot issues across the full stack including frontend, backend, middleware, data-layer, and routing.

• Amend and rewrite code in response to security findings.

• Document code using TDD, FDD, DFD, and SD methodologies.

• Create and automate CI/CD deployment pipelines for Teams bots, web applications, and Databricks bundles/notebooks through ADO.

• Incorporate validation steps such as Snyk scans and linting.

• Configure environments utilizing CI/CD pipelines.

• Collaborate with toxicologists and client project managers based on brief requirement descriptions.

• Deliver work as an individual contributor in short sprint cycles.

⛳️ Requirements

• Please send your resume to Careers.Apply@enabledata.com for a prompt response.

• Provide total years of experience.

• Specify relevant years of experience.

• Indicate current CTC.

• State expected CTC.

• Only immediate joiners will be considered.

🏝️ Benefits

• Opportunity to work with cutting-edge technologies.

• Collaborative and dynamic work environment.

• Competitive salary and benefits package.

• Opportunities for professional development and growth.