Databricks Platform Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop and sustain the Databricks workspace infrastructure across development, staging, and production environments.

• Architect, deploy, and manage Databricks platforms within Azure, AWS, or GCP cloud environments.

• Administer and enhance Databricks clusters, focusing on autoscaling, resource distribution, monitoring, and troubleshooting.

• Supervise the deployment and configuration of Databricks workspaces, including user management, access controls, and workspace policies.

• Partner with infrastructure and DevOps teams to establish CI/CD pipelines for Databricks workflows and notebooks.

• Oversee platform performance, logging, and system health management.

• Enforce data security and governance protocols in alignment with organizational policies and standards.

• Diagnose and resolve platform-related challenges in collaboration with cross-functional teams.

• Automate provisioning, upgrades, and scaling through Infrastructure-as-Code tools like Terraform or CloudFormation.

• Assist data engineers and data scientists in integrating Databricks with various data sources and tools.

• Keep abreast of the latest Databricks features, updates, and best practices.

⛳️ Requirements

• Over 5 years of experience in platform engineering, cloud infrastructure, or IT operations.

• More than 2 years of practical experience managing Databricks environments.

• In-depth knowledge of Databricks administration, including cluster configuration, workspace management, and job orchestration.

• Proficient in Apache Spark.

• Practical experience with Azure Government, AWS GovCloud, or GCP Public Sector.

• Skilled in Python, Bash, or PowerShell scripting and automation.

• Familiarity with Infrastructure-as-Code tools such as Terraform, CloudFormation, or ARM templates.

• Knowledgeable in cloud networking concepts, including VPCs, subnets, security groups, and private endpoints.

• Awareness of FedRAMP, FISMA, NIST 800-53, and CMMC standards.

• Experienced with RBAC, encryption both at rest and in transit, and audit logging.

• Capable of utilizing monitoring and logging tools like Datadog, AWS CloudWatch, or Azure Monitor.

• Proficient in diagnosing and resolving cluster performance issues, job failures, and other technical challenges.

• Databricks Certified Associate or Platform Administrator certification is preferred.

• Familiar with Unity Catalog administration and data governance is preferred.

• Experience in Databricks cost optimization, performance tuning, SSO, MFA, data retention, ETL/ELT, data warehousing, and real-time streaming technologies is preferred.

• Federal cloud certifications are preferred.

• Eligibility for Public Trust clearance or higher and prior experience with federal clients is preferred.

🏝️ Benefits

• 401(k) matching

• Dental insurance

• Health insurance

• Paid time off

• Parental leave

• Vision insurance