Principal Consultant – Databricks

This is a fully remote position, open to applicants in United States.

📋 Description

• Take ownership of the technical vision, target-state architecture, principles, standards, and roadmaps for intricate Azure Databricks projects.

• Facilitate discovery and architecture workshops, converting business priorities, security requirements, operational limitations, and success metrics into solution designs.

• Design secure and scalable enterprise lakehouse platforms encompassing ingestion, storage, transformation, orchestration, analytics, data sharing, and AI/ML enablement.

• Establish best practices for engineering, governance, identity, networking, CI/CD, observability, reliability, performance, and cost optimization.

• Oversee delivery by reviewing essential implementation artifacts, identifying risks, guiding technical decisions, and ensuring alignment between sold and delivered solutions.

• Assist in qualification, solution development, estimates, staffing plans, proposals, statements of work, demonstrations, and executive presentations.

• Create reusable reference architectures, accelerators, assessment methodologies, estimation models, delivery playbooks, and thought leadership content.

• Mentor architects, engineers, and consultants.

• Collaborate with Databricks, Microsoft, customer stakeholders, sales teams, and internal practices.

⛳️ Requirements

• Over 10 years of experience in enterprise data and analytics consulting.

• More than 5 years of experience in architecting and delivering production Databricks or Lakehouse solutions.

• At least 3 years in senior technical leadership, solution architecture, or principal-level consulting roles.

• Proven experience as the senior technical authority for complex enterprise data-platform engagements from discovery through production delivery.

• Extensive expertise in Azure Databricks covering platform architecture, data engineering, Unity Catalog governance, operationalization, performance, and cost management.

• Strong proficiency in Apache Spark, PySpark, Python, SQL, Delta Lake, batch and streaming patterns, ETL/ELT, data modeling, data quality, performance tuning, and troubleshooting in production environments.

• Familiarity with Microsoft Entra ID, Azure networking and private connectivity, ADLS, secrets and key management, monitoring, and related Azure data services.

• Experience with Databricks Partner Champion or Delivery Expert recognition, partner or Center of Excellence leadership, co-sell experience, or work in regulated and security-sensitive industries.

• Knowledge of Git-based development, automated testing, CI/CD, environment promotion, infrastructure as code, deployment automation, observability, runbooks, and support models.

• Strong consulting and business savvy linking architectural decisions to customer outcomes, delivery risks, scope, margins, adoption, and platform consumption.

• Excellent written, verbal, facilitation, and presentation skills suitable for executive-level communication.

• Experience in discovery, estimation, staffing, proposal or SOW development, technical pre-sales, delivery leadership, and guiding multidisciplinary teams.

• Self-motivated and collaborative leadership style with sound judgment, resilience, empathy, and the ability to create structure in uncertain situations.

• Databricks Certified Data Engineer Professional is highly preferred.

• Azure Platform Architect Badge and/or Databricks Data + AI Platform Administrator accreditation is preferred.

• Willingness to pursue an OZ-approved, role-aligned certification or accreditation path post-joining if lacking a relevant Databricks credential.

• Experience in large-scale platform modernization, cloud transformation, data warehouse migration, or consolidation involving Databricks is preferred.

• Background in creating reusable offerings, accelerators, reference architectures, assessment frameworks, delivery playbooks, or intellectual property is preferred.

• Experience in overseeing production analytics, machine-learning, or AI workloads beyond proof-of-concept delivery is preferred.

• Proficiency in Azure Databricks, Delta Lake, Unity Catalog, Databricks SQL, Lakeflow Jobs and pipelines, Apache Spark, PySpark, Python, SQL, Microsoft Azure, Microsoft Entra ID, Azure Storage/ADLS, Azure networking, Private Link, Key Vault, Azure Monitor, Git, GitHub and/or Azure DevOps, automated testing, CI/CD, Terraform or comparable infrastructure as code, Databricks CLI and APIs, and Declarative Automation Bundles.

• Exposure to MLflow, model serving, Power BI, Microsoft Fabric, Azure Data Factory, and Microsoft Purview is preferred.

🏝️ Benefits

• Competitive Pay

• 401K

• Health Coverage

• Unlimited PTO

• Flexible work arrangements, including hybrid and/or remote options

• Tools and support to excel

• A collaborative culture that emphasizes growth and work-life balance