
AI Architect – Generative AI, Databricks
Posted 11 hours ago

Posted 11 hours ago
This is a fully remote position, open to applicants in Poland, +2 more countries.
• Design and manage the comprehensive GenAI Lakehouse architecture on Databricks, converting unstructured email discussions and intricate attachments into verified, ERP-compatible tabular data.
• Create secure, incremental ingestion pipelines utilizing Microsoft Graph API and PySpark parsing engines for PDFs, Excel files, and HTML content.
• Direct deterministic extraction workflows on Databricks Model Serving by employing Claude/GPT along with precise Pydantic schemas.
• Develop a three-tier anti-hallucination verification system featuring schema constraints, source citations, and PySpark string validations.
• Design and implement a Human-in-the-Loop review application on Databricks Apps using Streamlit/Dash, incorporating reconciliation, manual overrides, and SSO/OBO authentication.
• Model Gold-layer pricing staging tables and create automated batch feeds for ERP usage through Databricks SQL Warehouses and REST APIs.
• Establish governance policies for Unity Catalog, including data lineage, error-quarantine procedures, and system security measures.
• Produce architectural documentation, data models, and interface specifications.
• Mentor client engineering teams.
• Collaborate with Databricks product teams and client stakeholders.
• Bachelor’s or Master’s degree in Computer Science, Artificial Intelligence, Data Engineering, or a related discipline.
• Over 7 years of experience in data engineering, data architecture, or applied machine learning systems.
• At least 4 years of practical experience in designing and implementing solutions on the Databricks Lakehouse Platform, including Unity Catalog, Delta Lake, Databricks Model Serving, and Databricks SQL.
• Proven experience as an Applied GenAI / LLMOps Engineer/Architect, constructing structured extraction pipelines using function calling, Pydantic validation, and custom guardrail/citation-verification systems.
• Established track record of parsing messy, semi-structured, and unstructured business documents using Python and PySpark libraries, such as PyMuPDF, pdfplumber, OpenPyXL, and BeautifulSoup4.
• Hands-on experience in developing and deploying interactive data applications using Streamlit or Dash, including containerized apps natively within Databricks Apps.
• Strong knowledge of Microsoft Graph API, OAuth 2.0 client credential flows, Microsoft Entra ID (Azure AD), and Databricks Secret Scopes.
• Solid understanding of data modeling, governance, and auditability within Unity Catalog, covering Volumes, Delta Tables, ACLs, and row/column-level security.
• Familiarity with downstream enterprise ERP integration patterns.
• Excellent problem-solving and analytical abilities.
• Outstanding communication, consulting, and client-facing skills.
• 100% Remote: Full flexibility to work from any location within Poland, Romania, or Greece.
• All engineers receive an Apple MacBook Air M4 15".
• Referral bonus available for bringing other top-tier engineers to the team.
• Coverage for all Databricks technical certifications.
• Support to achieve the highest levels of Databricks expertise, including the Champion program.
• Opportunities to present at global industry conferences and contribute to technical thought leadership.
• Direct collaboration with Databricks MVPs and core product teams.
• Employment Contract (UoP) or B2B options available in Poland.
• B2B cooperation available for candidates in Romania and Greece.
OpenAI
Codat
PagBank
Life360
Get handpicked remote jobs straight to your inbox weekly.