Principal Consultant – Databricks

This is a fully remote position, open to applicants in United States.

📋 Description

• Take ownership of the technical vision, target-state architecture, principles, standards, and roadmaps for intricate Azure Databricks projects.

• Facilitate discovery and architecture workshops, converting business priorities, security requirements, operational limitations, and success criteria into solution designs.

• Design secure and scalable enterprise Lakehouse platforms that encompass ingestion, storage, transformation, orchestration, analytics, data sharing, and AI/ML enablement.

• Define standards for engineering, governance, identity, networking, CI/CD, observability, reliability, performance, and cost optimization.

• Oversee delivery by reviewing implementation artifacts, identifying risks, guiding technical choices, and ensuring alignment between sold and delivered solutions.

• Assist in qualification, solution shaping, estimations, staffing plans, proposals, statements of work, demonstrations, and executive presentations.

• Create reference architectures, accelerators, assessment methods, estimation models, delivery playbooks, and provide thought leadership.

• Mentor architects, engineers, and consultants.

• Collaborate with Databricks, Microsoft, customer stakeholders, sales teams, and internal practices.

• Act as the senior technical authority for enterprise data-platform projects from discovery through to production delivery.

⛳️ Requirements

• Over 10 years of experience in enterprise data and analytics consulting.

• At least 5 years of experience architecting and delivering production Databricks or Lakehouse solutions.

• A minimum of 3 years in senior technical leadership, solution architecture, or principal-level consulting roles.

• At least 3 years of experience within an insurance firm in a consulting role.

• Proven experience as the senior technical authority for complex enterprise data-platform engagements from discovery to production delivery.

• Extensive expertise in Azure Databricks, including platform architecture, data engineering, Unity Catalog governance, operationalization, performance, and cost management.

• Strong proficiency in Apache Spark, PySpark, Python, SQL, Delta Lake, batch and streaming patterns, ETL/ELT, data modeling, data quality, performance tuning, and troubleshooting in production environments.

• Familiarity with Microsoft Entra ID, Azure networking and private connectivity, ADLS, secrets and key management, monitoring, and related Azure data services.

• Experience with Git-based development, automated testing, CI/CD, environment promotion, infrastructure as code, deployment automation, observability, runbooks, and support models.

• Exceptional consulting and business acumen to connect architecture decisions with customer outcomes, delivery risks, scope, margin, adoption, and platform consumption.

• Excellent written, verbal, facilitation, and presentation skills at the executive level.

• Experience in discovery processes, estimation, staffing, proposal or SOW development, technical pre-sales, delivery leadership, and guiding multidisciplinary teams.

• Self-directed and collaborative leader with sound judgment, resilience, empathy, and the ability to create structure in ambiguous situations.

• Databricks Certified Data Engineer Professional is strongly preferred.

• Azure Platform Architect Badge and/or Databricks Data + AI Platform Administrator accreditation is preferred.

• Recognition as a Databricks Partner Champion or Delivery Expert, leadership in partner or Center of Excellence, experience in co-selling, or work in regulated and security-sensitive industries is preferred.

• Willingness to pursue an OZ-approved, role-aligned certification or accreditation path post-joining if lacking an appropriate Databricks credential.

• Databricks Certified Generative AI Engineer Associate or Machine Learning Professional is a plus.

• Experience with large-scale platform modernization, cloud transformation, data warehouse migration, or consolidation involving Databricks is preferred.

• Proven experience in creating reusable offerings, accelerators, reference architectures, assessment frameworks, delivery playbooks, or intellectual property is preferred.

• Experience overseeing production analytics, machine-learning, or AI workloads beyond proof-of-concept delivery is preferred.

• Familiarity with Azure Databricks, Delta Lake, Unity Catalog, Databricks SQL, Lakeflow Jobs and pipelines, Apache Spark, PySpark, Python, SQL, Microsoft Azure, Microsoft Entra ID, Azure Storage/ADLS, Azure networking, Private Link, Key Vault, Azure Monitor, Git, GitHub and/or Azure DevOps, automated testing, CI/CD, Terraform or comparable infrastructure as code, Databricks CLI and APIs, and Declarative Automation Bundles.

• Exposure to MLflow, model serving, Power BI, Microsoft Fabric, Azure Data Factory, and Microsoft Purview is preferred.

🏝️ Benefits

• Competitive Pay

• 401K

• Health Coverage

• Unlimited PTO

• Flexible work arrangements, including hybrid and/or remote work

• Tools and support to thrive

• A culture of collaboration, growth, and work-life balance