Azure Databricks Architect

This is a fully remote position, open to applicants in United States.

📋 Description

• Architect and design enterprise-level data platforms utilizing Azure Databricks and Azure cloud services;

• Define and implement Lakehouse architecture patterns by employing Delta Lake and Databricks best practices;

• Create scalable frameworks for data ingestion, transformation, and serving that support both batch and streaming workloads;

• Design and execute solutions for real-time and near real-time data processing using Structured Streaming, CDC, Change Data Feed (CDF), and event-driven architectures;

• Design and manage multi-workspace Databricks environments along with shared Metastore architectures.

• Establish governance for the platform, including security controls, cluster policies, RBAC models, and access management frameworks;

• Design and implement CI/CD pipelines for Databricks assets, infrastructure, and data workloads.

• Automate infrastructure provisioning through Infrastructure as Code (Terraform, ARM, or Bicep);

• Develop strategies for deployment and release management across development, staging, and production environments;

• Optimize Databricks workspaces, clusters, jobs, and storage configurations for improved performance and cost efficiency;

• Implement solutions for monitoring, alerting, logging, and observability in platform operations;

• Define and enforce standards for data architecture, data quality frameworks, and governance policies;

• Integrate Azure Databricks with Azure Data Factory, Azure Synapse, Azure Storage, Azure Key Vault, Azure Entra ID, and other Azure services;

• Support Lakeflow Connect, managed connectors, CDC implementations, and interoperability within the enterprise data ecosystem;

• Collaborate with engineering and business teams to convert requirements into scalable technical solutions;

• Provide architectural guidance, technical leadership, and mentorship to engineering teams.

⛳️ Requirements

• 7+ years of experience in Data Engineering, Data Platform Engineering, Cloud Data Architecture, or Data Platform Consulting.

• 3+ years of direct experience architecting solutions with Azure Databricks.

• Strong expertise in Apache Spark, PySpark, Delta Lake, and Databricks Lakehouse architecture.

• Practical experience with Unity Catalog, data governance, lineage, permissions models, and catalog isolation strategies.

• Experience in designing and managing Databricks Metastore and multi-workspace architectures.

• Strong knowledge of Databricks administration, user provisioning, RBAC, workspace management, and security controls.

• Extensive experience in designing cloud-native solutions on Microsoft Azure.

• Solid understanding of Azure Data Factory, Azure Storage, Azure Key Vault, Azure Monitor, Azure Entra ID, and Azure networking concepts.

• Experience in designing streaming and low-latency data pipelines using Databricks Structured Streaming.

• Practical experience with CDC, Change Data Feed (CDF), and modern data ingestion patterns.

• Experienced in building and maintaining CI/CD pipelines using Azure DevOps and/or GitHub Actions.

• Familiarity with implementing Infrastructure as Code using Terraform, ARM Templates, or Bicep.

• Strong understanding of DevOps principles, deployment automation, and environment management.

• Proficient in SQL and Python programming.

• Excellent communication, stakeholder management, and consulting skills.

🏝️ Benefits

• Digital-First Approach: Exceptional talent knows no boundaries! You can work from anywhere — we hire and collaborate with professionals globally.

• Remote Work Model: Achieve a balance between your professional and personal life with our flexible working conditions, allowing you to perform your best from any location.

• Exciting Projects: Engage in impactful projects across various industries that challenge and inspire creativity.

• Boost Your Expertise: Advance your career through continuous learning, development opportunities, and hands-on experience.

• Join the Best Team Ever: Collaborate with a diverse and cross-cultural team of passionate technologists and creative thinkers.