Senior AI Data Engineer

This is a fully remote position, open to applicants in Colombia.

📋 Description

• Design and construct medallion lakehouses utilizing Databricks, Microsoft Fabric, and Snowflake.

• Take ownership of data ingestion from ERPs, CRMs, and SaaS platforms, including handling change data capture and schema drift.

• Model and transform data using dbt and orchestrate pipelines with Airflow, Dagster, or Lakeflow Jobs.

• Deploy pipelines through CI/CD processes utilizing GitHub Actions and Declarative Automation Bundles.

• Integrate data quality checks and data contracts within the pipelines.

• Configure and manage coding agents in client Databricks repositories.

• Construct production RAG pipelines that involve parsing, OCR, chunking, embeddings, hybrid retrieval, and reranking.

• Design GraphRAG systems leveraging knowledge graphs and graph stores.

• Select and justify retrieval architectures based on specific use cases, costs, and maintenance considerations.

• Operate vector, full-text, and hybrid search infrastructure with incremental indexing and permission-aware retrieval.

• Develop and deploy retrieval-backed agents on Databricks Agent Bricks.

• Establish governed semantic and metric definitions across dbt, Power BI, Cube, Databricks, and Snowflake.

• Lead data readiness assessments and document business logic, identify gaps, and propose remediation strategies.

• Expose governed data to agents through MCP.

• Create golden datasets, evaluation pairs, and leakage-free backtesting splits.

• Instrument retrieval and agent pipelines, tracking recall, precision, groundedness, experiments, and model versions.

• Monitor for drift, freshness, and gaps in production data.

• Implement RBAC, row-level security, and data masking across data and retrieval systems.

• Ensure data pathways satisfy SOC 2 and client AI governance requirements.

• Collaborate with client IT, finance, and operations leaders on project scope, access, and build-versus-buy decisions.

• Document architecture decisions and contribute reusable engineering components to Cadre’s internal library.

⛳️ Requirements

• Over 5 years of experience in data engineering.

• Proven experience with at least one production lakehouse designed and operated on Databricks or Microsoft Fabric using medallion architecture.

• A Master’s degree or PhD in Computer Science, Mathematics, Engineering, or a related quantitative discipline.

• Experience with production RAG pipelines, including hybrid retrieval, reranking, chunking, and permission-aware indexing.

• Background in designing and implementing GraphRAG or knowledge-graph retrieval systems.

• Hands-on experience orchestrating production data pipelines using Apache Airflow or Dagster.

• Proficient in SQL, with strong skills in Python and PySpark.

• Production experience with dbt or platform-native pipelines and ingestion tools.

• Daily usage and review of coding agents, such as Claude Code.

• Experience in authoring semantic or metrics layers.

• Working knowledge of machine learning principles, including data leakage, time-based validation, class imbalance, precision, and recall.

• Experience in building datasets for model or retriever training and evaluation.

• Familiarity with tools like Langfuse, Arize Phoenix, Braintrust, or MLflow and MLflow Registry.

• Practical experience in building Databricks agents using Agent Bricks or the Mosaic AI Agent Framework.

• Experience managing Claude Code or Codex under governed environments.

• Knowledge of Omnigent, Genie Migrate, Agent Bricks CLI, RAG, text-to-SQL, and MCP.

• Governance experience with Unity Catalog, Purview, or Snowflake RBAC.

• Experience in client-facing consulting or forward-deployed roles.

• Proficient in professional English.

• Strong plus: Familiarity with Snowflake Cortex, Palantir Foundry ontologies, RDF/OWL knowledge graphs, DSPy prompt optimization, or LoRA fine-tuning.

🏝️ Benefits

• An AI-native culture.

• Access to cutting-edge models and capabilities through partnerships with Anthropic and OpenAI.

• Opportunities for growth and the ability to transition into new teams and initiatives.

• A bureaucracy-free environment with clear accountability and an idea-driven culture.