Databricks Platform Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, implement, oversee, and provide support for enterprise Databricks environments tailored for a federal client.

• Set up and configure Databricks workspaces from scratch utilizing Terraform and the Databricks Secure Reference Architecture across development, testing, and production settings.

• Facilitate Databricks deployments in both AWS Commercial and AWS GovCloud environments.

• Architect and execute AWS networking solutions, which include VPCs, subnets, PrivateLink, Transit Gateway, security groups, and DNS.

• Oversee connectivity for Databricks control-plane and data-plane.

• Manage Unity Catalog, covering metastores, catalogs, schemas, external locations, permissions, grants, as well as row- and column-level security.

• Develop and sustain CI/CD pipelines that support infrastructure deployment and Databricks Asset Bundles.

• Create platform governance standards, architectural guidelines, tagging strategies, naming conventions, and operational best practices.

• Automate the provisioning and management of cluster policies, secrets, service principals, SSO, and SCIM integrations.

• Monitor the health, utilization, performance, and costs associated with the platform.

• Spot opportunities for rightsizing, cost reduction, and enhanced platform efficiency.

• Act as a key escalation point for platform incidents, troubleshooting, and conducting root-cause analysis.

• Collaborate with both technical and business stakeholders on architectural, governance, security, and operational decisions related to the platform.

⛳️ Requirements

• An active Secret security clearance or higher is mandatory.

• U.S. citizenship is required.

• A minimum of 6 years of experience in engineering, administering, or supporting Databricks platforms.

• Robust hands-on experience with Databricks in AWS environments.

• Advanced expertise in Terraform for networking, IAM, compute, and storage.

• In-depth knowledge of the Databricks Secure Reference Architecture.

• Practical experience in administering Unity Catalog.

• Familiarity with Databricks Asset Bundles or similar deployment automation frameworks.

• Strong grasp of CI/CD practices and infrastructure-as-code methodologies.

• Proficient in Python programming.

• Experience in Linux systems administration.

• Hands-on experience with AWS services including VPC, IAM, S3, KMS, EC2, and CloudWatch.

• Excellent troubleshooting and root-cause analysis skills.

• Strong communication abilities, with the capacity to guide stakeholders on technical and platform-related decisions.

• Preferred: Experience with AWS GovCloud.

• Preferred: Background in regulated, security-sensitive, or compliance-focused environments.

• Preferred: Experience with FISMA High, FedRAMP High, or similar federal security standards.

• A Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field is preferred.

🏝️ Benefits

• Flexible remote work options.

• Chance to contribute to federal government programs.

• Experience working with AWS Commercial and AWS GovCloud environments.

• Engagement in regulated, security-sensitive, and compliance-heavy environments.