Databricks Platform Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee the Databricks platform across various environments, including workspaces, Unity Catalog, cluster policies, budget management, and platform guardrails.

• Administer identity and access management utilizing Microsoft Entra ID single sign-on, account-level SCIM group provisioning, service principals, and scalable access control models.

• Collaborate with IT Infrastructure and Security teams to manage networking and security, encompassing customer-managed virtual private clouds, private connectivity, encryption, egress controls, audit logs, and system tables integrated with enterprise monitoring tools.

• Handle infrastructure and workloads through Terraform, Databricks Asset Bundles, and CI/CD methodologies.

• Develop and uphold cost-governance practices, including tagging standards, business-unit showback, budget oversight, and recommendations for spending reduction.

• Offer guidance to data scientists, data engineers, analysts, and business stakeholders on platform architecture and solution design.

• Provide actionable recommendations that balance organizational standards with business requirements.

• Assess emerging capabilities of the Databricks platform and formulate adoption strategies.

• Monitor platform updates and evaluate serverless computing, Unity Catalog, Lakeflow, and other new features.

• Convert new functionalities into practical, maintainable recommendations.

• Influence the platform’s scalability, security, governance, and performance through architectural decisions.

⛳️ Requirements

• Several years of practical experience in administering and architecting Databricks environments in production settings.

• Extensive, up-to-date experience in Databricks administration on AWS, or substantial Azure Databricks experience with a reliable path to supporting AWS.

• Proven experience in designing and managing Unity Catalog within a multi-workspace framework.

• Demonstrated expertise in developing catalog structures and access-control models for expanding organizations.

• Experience in integrating Databricks with Microsoft Entra ID, covering aspects like single sign-on and SCIM provisioning.

• Strong background in Terraform and CI/CD for infrastructure and platform management.

• Familiarity with cloud networking concepts, including virtual private clouds, private connectivity, encryption, egress controls, and security-review procedures.

• Ability to collaborate with IT Infrastructure and Security teams, articulate platform requirements, adhere to enterprise standards, and advocate for architectural enhancements.

• Sound judgment concerning platform defaults and organizational needs.

• Exceptional written and verbal communication skills.

• Experience in supporting government, military, or controlled-data environments, encompassing Controlled Unclassified Information and Cybersecurity Maturity Model Certification requirements (preferred).

• Experience with enclave separation, controlled environments, and regulated-data operations (preferred).

• Databricks certifications, such as Databricks Certified Data Engineer Professional or Databricks platform administration credentials (nice to have).

• Familiarity with Databricks Asset Bundles (nice to have).

• Experience in a federated, decentralized, or multi-entity organization (nice to have).

• Experience with MLflow, machine learning operations, model serving, and related Databricks capabilities (nice to have).

• Travel of up to 20% may be required.

🏝️ Benefits

• Performance-based bonuses.

• Equity participation.

• Medical insurance.

• Dental insurance.

• Vision insurance.

• 401(k) with company match.

• Paid time off.

• Company holidays.

• Wellness programs.

• Employee-assistance resources.

• Support for professional development.