Senior Databricks Administrator – Architect

This is a fully remote position, open to applicants in Spain.

📋 Description

• Collaborate with the Data Platforms and Integration team to design and implement scalable enterprise data platform solutions using Azure Databricks, Delta Lake, Unity Catalog, Azure Data Lake Storage Gen2, Azure Data Factory, Azure Event Hub, Azure SQL, and other related Azure services;

• Create and develop enterprise data engineering frameworks that support batch processing, streaming, ELT/ETL, and real-time analytics workloads;

• Manage and maintain Azure Databricks workspaces, including cluster configuration, job scheduling, workspace administration, Unity Catalog implementation, secret management, compute policies, cluster policies, and workspace governance;

• Define and establish enterprise security architecture, incorporating identity integration with Microsoft Entra ID (Azure AD), Role-Based Access Control (RBAC), Privileged Identity Management (PIM), data encryption, private networking, managed identities, Key Vault integration, and secure access methods;

• Set up enterprise data governance standards through Unity Catalog, addressing data lineage, data quality, metadata management, catalog organization, access policies, and auditing capabilities;

• Deploy and configure CI/CD pipelines using Azure DevOps and Databricks Asset Bundles (DAB), Infrastructure as Code (Terraform/Bicep where relevant), deployment automation, release management, and environment promotion strategies;

• Enhance Databricks workloads via performance tuning, Spark optimization, cluster sizing, workload isolation, caching strategies, partitioning, Delta optimization, Photon, and cost optimization best practices;

• Design enterprise monitoring, logging, operational dashboards, alerting, and platform health monitoring using Azure Monitor, Log Analytics, Application Insights, and Databricks system tables;

• Develop reusable architecture patterns, reference implementations, development standards, naming conventions, coding standards, operational procedures, and platform documentation;

• Assist in migrating legacy enterprise data warehouses, ETL solutions, and reporting platforms to Azure Databricks and contemporary Lakehouse architecture;

• Provide technical support and troubleshooting for production incidents related to Azure Databricks, Spark workloads, data pipelines, and platform infrastructure;

⛳️ Requirements

• A university degree combined with 9 years of relevant IT professional experience;

• A minimum of 8 years of experience in designing and implementing enterprise data platforms;

• At least 5 years of hands-on experience with Azure Databricks in enterprise environments;

• Experience in architecting modern Lakehouse solutions utilizing Azure Databricks and Delta Lake;

• Proficient in administering Azure Databricks workspaces, clusters, Unity Catalog, compute policies, job orchestration, and workspace security;

• Demonstrated experience in developing data engineering solutions using Apache Spark (PySpark and/or Spark SQL);

• Experienced in implementing enterprise data governance through Unity Catalog, metadata management, data lineage, and fine-grained access controls;

• Familiarity with Infrastructure as Code using Terraform and/or Bicep;

• Strong experience in implementing CI/CD pipelines with Azure DevOps, automated testing, release management, and deployment automation;

• Experience in designing and working with enterprise security architectures, including RBAC, Managed Identities, Private Endpoints, Key Vault integration, encryption, and network security;

• Experience in establishing enterprise data quality frameworks, monitoring, observability, and operational dashboards;

• Solid understanding of data modeling techniques, including dimensional modeling, Data Vault, medallion architecture, and enterprise data warehouse concepts;

• Profound knowledge of Azure Data Lake Storage Gen2, Azure Data Factory, Azure Event Hub, Azure SQL Database, Azure Key Vault, Azure Monitor, Microsoft Entra ID (Azure AD), and Azure networking;

• Strong SQL programming skills, along with proficiency in Python and PySpark development;

• Knowledge of AI/ML workloads, Databricks ML, Feature Store, Model Serving, and Generative AI architectures is an asset;

• Experience with Power BI, Microsoft Fabric, Synapse Analytics, or other enterprise analytics platforms is highly desirable;

• Background working within large international organizations, NGOs, or United Nations agencies is preferred;

• The following certifications are beneficial: Databricks Certified Platform, Architecture/Microsoft Azure Solution Architecture Expert/Databricks, Certified Data Engineer Associate/Professional;

• Excellent command of the English language;

🏝️ Benefits

• If you are looking for a career in an exciting and dynamic company where you will contribute as part of a team for a major European Institution, operating in an international, multilingual, and multicultural environment with genuine opportunities to make an impact, please send us your detailed CV in English, referencing **(20199/07/2026).**

• We offer competitive remuneration (either on a contract basis or with a full benefits package) based on qualifications and experience. All applications will be treated confidentially.