Principal Databricks Architect

This is a fully remote position, open to applicants in Ohio.

📋 Description

• Define and advocate for comprehensive architecture for the Databricks data science and AI platform across a global structure.

• Create reusable reference architectures, design patterns, and blueprints.

• Lead the governance of Unity Catalog, focusing on catalog hierarchies, fine-grained access, data masking, lineage, and classification.

• Collaborate with infrastructure, security, and compliance teams to implement least-privileged access, responsible AI practices, regulatory compliance, and data sovereignty.

• Design low-duplication integration between Databricks and Snowflake utilizing Apache Iceberg and Unity Catalog federation.

• Facilitate bi-directional data access and feature sharing across various platforms.

• Develop self-service frameworks, templates, IDE/notebook experiences, and documentation for actuarial, data science, and analytics teams.

• Enhance ModelOps throughout the model lifecycle, governance, versioning, and deployment using MLflow, Model Serving, and the AI Agent Framework.

• Optimize compute resources, clusters, and job performance for reliability and cost efficiency.

• Implement observability, tagging, chargeback/showback, and self-service cost and usage analytics.

• Act as a trusted advisor through discovery workshops, proofs-of-concept, and technology assessments.

• Curate a community of practice and promote endorsed Databricks patterns.

• Mentor and enhance the skills of team members while building high-performing cross-functional data and AI teams.

⛳️ Requirements

• Bachelor’s Degree in Arts/Sciences (BA/BS) in Computer Science or equivalent education and experience — Required.

• Over 10 years of progressive experience in enterprise-level IT functions — Required.

• More than 7 years of experience managing large enterprise data initiatives and teams — Required.

• At least 5 years of experience architecting and operating enterprise-grade data and AI/ML platforms, with hands-on experience in Databricks and Apache Spark on a major cloud; AWS preferred, Azure or GCP accepted — Required.

• Extensive knowledge of the Databricks Data Intelligence Platform, including Unity Catalog, Delta Lake/Iceberg, Lakeflow/Delta Live Tables, MLflow, Model Serving, AI Agent Framework/Agent Bricks, Lakebase, and Genie — Required.

• Understanding of lakehouse architecture, ETL/ELT processes, Terraform, and Databricks Asset Bundles — Required.

• Proven success in building self-service platform capabilities and enhancing developer experiences for data science teams.

• ModelOps experience that includes lifecycle management, versioning, governance, and deployment automation — Required.

• Exceptional consulting, stakeholder management, and communication skills, with a track record of influencing senior leadership and driving technology adoption — Required.

• Ability to foster innovation, implement modern practices, and lead technological transformations — Required.

• Capability to articulate and communicate architecture vision, roadmaps, and best practices to both technical and non-technical audiences — Required.

• Familiarity with integrating Databricks with Snowflake and other engines through Apache Iceberg and Unity Catalog/Lakehouse federation — Preferred.

• Willingness to participate in cross-time-zone meetings — Preferred.

🏝️ Benefits

• Annual bonus plan.

• Eligibility for long-term equity incentive plans for select positions.

• Comprehensive health benefits.

• Retirement benefits.

• Additional employee benefits.

• A respectful and inclusive work environment.

• Opportunities for career advancement and learning from diverse colleagues.