Databricks Platform Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee the Databricks platform across various environments, which includes workspaces, Unity Catalog, cluster policies, budget policies, and platform guardrails.

• Manage identity and access through Microsoft Entra ID single sign-on, account-level SCIM group provisioning, service principals, and scalable access-control frameworks.

• Collaborate with IT Infrastructure and Security on networking and security aspects, such as customer-managed VPCs, private connectivity, encryption, egress controls, audit logs, and system tables.

• Handle infrastructure and workloads as code by utilizing Terraform, Databricks Asset Bundles, and CI/CD practices.

• Implement cost-governance strategies that include tagging standards, business-unit showback, budget controls, and recommendations for spending reduction.

• Provide guidance to data scientists, data engineers, analysts, and business stakeholders regarding platform architecture and solution design.

• Assess emerging capabilities within Databricks and determine the best approach for their adoption.

• Monitor platform updates and convert new features such as serverless computing, Unity Catalog, and Lakeflow into actionable recommendations.

• Assist in architecture reviews, security and governance discussions, engage with business stakeholders, and promote team collaboration.

⛳️ Requirements

• Extensive hands-on experience in administering and architecting production Databricks environments.

• Profound, up-to-date, production-level Databricks administration experience on AWS, or significant Azure Databricks experience with a credible pathway to supporting AWS.

• Experience in designing and managing Unity Catalog within a multi-workspace setting.

• Proven experience in developing catalog structures and access-control models for a growing organization.

• Proficiency in integrating Databricks with Microsoft Entra ID, including single sign-on and SCIM provisioning.

• Strong background in Terraform and CI/CD practices for managing infrastructure and platform.

• Familiarity with cloud networking, encompassing VPCs, private connectivity, encryption, egress controls, and security-review processes.

• Capability to collaborate with IT Infrastructure and Security teams while constructively advocating for architectural enhancements.

• Sound judgment in relation to platform defaults and organizational requirements.

• Excellent written and verbal communication abilities.

• Experience in supporting government, military, or controlled-data environments is preferred.

• Databricks certifications, Databricks Asset Bundles, federated organizations, MLflow, machine learning operations, and model serving are advantageous.

• Willingness to travel up to 20% may be required.

🏝️ Benefits

• Performance-based bonuses.

• Equity participation.

• Medical, dental, and vision insurance.

• 401(k) with company match.

• Paid time off and company holidays.

• Wellness programs and employee-assistance resources.

• Professional development support.