Solutions Architect

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee the complete deployment process of AI applications on the Databricks Apps platform, from the initial architecture design to production rollout and ongoing support.

• Design serverless-first solutions utilizing Databricks Apps along with its integrated security, compliance, and resource management features.

• Develop and integrate AI agents, including Retrieval-Augmented Generation (RAG) patterns, using Databricks Agent tools.

• Manage the entire lifecycle of AI agents: including creation, deployment, testing, and performance monitoring.

• Implement data governance utilizing Unity Catalog for centralized access control, data lineage, and security measures.

• Connect applications with Databricks SQL to enable efficient querying of large-scale datasets.

• Ensure secure authentication through OAuth and platform-specific identity and authorization methodologies.

• Set up and enhance Model Serving and Serving Endpoints for machine learning models and large language model agents.

• Collaborate with data engineering, security, and business stakeholders to interpret requirements and create scalable, governed, production-ready solutions.

• Establish architectural standards, reference patterns, and best practices for developing AI applications on Databricks.

• Partner with higher education clients and CampusWorks colleagues to assess processes, design future-state workflows, and architect, develop, test, train, and implement process and technical solutions.

⛳️ Requirements

• Extensive expertise in Python for developing applications, agents, and pipelines.

• Proven experience in designing and optimizing Databricks SQL queries for large datasets.

• Background in implementing secure authentication and authorization processes using OAuth / IAM.

• Proficient in deploying and scaling machine learning models and large language model agents through Model Serving and serving endpoints.

• Familiarity with AI agent frameworks and RAG architectures, such as LangChain and/or LlamaIndex.

• Experience in logging, testing, debugging, and monitoring agents and models utilizing MLflow.

• Knowledge of application frameworks such as FastAPI, Flask, Streamlit, Dash, or Gradio.

• Front-end development experience using React or a similar modern JavaScript framework.

• Skilled in utilizing Databricks Jobs and workflow orchestration for data pipelines and ETL processes.

• Understanding of Microsoft Azure networking, security, and compliance principles.

• Strong architectural decision-making skills that balance scalability, security, governance, and cost considerations.

• Capability to lead technical delivery and provide mentorship to engineering teams.

• Effective communication skills with both technical and executive/business stakeholders.

• Previous experience in delivering enterprise chatbot or conversational AI solutions is preferred.

• Familiarity with token streaming, request logging, and review-app patterns for production AI agents is preferred.

• Relevant certifications such as Databricks Certified Data Engineer / ML or Azure Solutions Architect are preferred.

🏝️ Benefits

• Flexibility to work remotely.

• Engagement in project-based consulting.

• Opportunity to make a meaningful contribution to leading higher education institutions.