AI Architect – Generative AI, Databricks

Posted 11 hours ago

This is a fully remote position, open to applicants in Poland, +2 more countries.

📋 Description

• Design and manage the comprehensive GenAI Lakehouse architecture on Databricks, converting unstructured email discussions and intricate attachments into verified, ERP-compatible tabular data.

• Create secure, incremental ingestion pipelines utilizing Microsoft Graph API and PySpark parsing engines for PDFs, Excel files, and HTML content.

• Direct deterministic extraction workflows on Databricks Model Serving by employing Claude/GPT along with precise Pydantic schemas.

• Develop a three-tier anti-hallucination verification system featuring schema constraints, source citations, and PySpark string validations.

• Design and implement a Human-in-the-Loop review application on Databricks Apps using Streamlit/Dash, incorporating reconciliation, manual overrides, and SSO/OBO authentication.

• Model Gold-layer pricing staging tables and create automated batch feeds for ERP usage through Databricks SQL Warehouses and REST APIs.

• Establish governance policies for Unity Catalog, including data lineage, error-quarantine procedures, and system security measures.

• Produce architectural documentation, data models, and interface specifications.

• Mentor client engineering teams.

• Collaborate with Databricks product teams and client stakeholders.


⛳️ Requirements

• Bachelor’s or Master’s degree in Computer Science, Artificial Intelligence, Data Engineering, or a related discipline.

• Over 7 years of experience in data engineering, data architecture, or applied machine learning systems.

• At least 4 years of practical experience in designing and implementing solutions on the Databricks Lakehouse Platform, including Unity Catalog, Delta Lake, Databricks Model Serving, and Databricks SQL.

• Proven experience as an Applied GenAI / LLMOps Engineer/Architect, constructing structured extraction pipelines using function calling, Pydantic validation, and custom guardrail/citation-verification systems.

• Established track record of parsing messy, semi-structured, and unstructured business documents using Python and PySpark libraries, such as PyMuPDF, pdfplumber, OpenPyXL, and BeautifulSoup4.

• Hands-on experience in developing and deploying interactive data applications using Streamlit or Dash, including containerized apps natively within Databricks Apps.

• Strong knowledge of Microsoft Graph API, OAuth 2.0 client credential flows, Microsoft Entra ID (Azure AD), and Databricks Secret Scopes.

• Solid understanding of data modeling, governance, and auditability within Unity Catalog, covering Volumes, Delta Tables, ACLs, and row/column-level security.

• Familiarity with downstream enterprise ERP integration patterns.

• Excellent problem-solving and analytical abilities.

• Outstanding communication, consulting, and client-facing skills.


🏝️ Benefits

• 100% Remote: Full flexibility to work from any location within Poland, Romania, or Greece.

• All engineers receive an Apple MacBook Air M4 15".

• Referral bonus available for bringing other top-tier engineers to the team.

• Coverage for all Databricks technical certifications.

• Support to achieve the highest levels of Databricks expertise, including the Champion program.

• Opportunities to present at global industry conferences and contribute to technical thought leadership.

• Direct collaboration with Databricks MVPs and core product teams.

• Employment Contract (UoP) or B2B options available in Poland.

• B2B cooperation available for candidates in Romania and Greece.

People also viewed

OpenAI10 hours ago

Applied AI Architect

US flagUnited States OnlyFull-timeAI Engineer$156.6k – $220.5k/year
ApplyView job
Codat11 hours ago

Senior AI Engineer

GB flagUnited Kingdom OnlyFull-timeAI Engineer£98k – £115k/year
ApplyView job
PagBank11 hours ago

Artificial Intelligence Engineer, Mid-Level

BR flagBrazil OnlyFull-timeAI Engineer
ApplyView job
Life36011 hours ago

Senior Firmware Engineer, AI Native

US flagUnited States, +1 more countryFull-timeAI Engineer$109.5k – $199.5k/year
ApplyView job
CINC Systems12 hours ago

Staff Applied AI Engineer

Anywhere in the WorldFull-timeAI Engineer
ApplyView job
BM Digital12 hours ago

AI Engineer

PH flagPhilippines, +1 more countryFull-timeAI Engineer$24k – $48k/year
ApplyView job

Never miss a great job!

Get handpicked remote jobs straight to your inbox weekly.

Trusted by 7,400+ designers