Senior Databricks Platform Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, configure, and support the enterprise Databricks Lakehouse Platform, which includes workspaces, Unity Catalog, and scalable data architectures.

• Administer and enhance Databricks compute resources, clusters, SQL warehouses, serverless capabilities, and workload management.

• Develop and refine data pipelines, ETL/ELT processes, and ingestion frameworks utilizing Apache Spark, Delta Lake, Delta Live Tables, and Databricks Workflows.

• Create, uphold, and govern Unity Catalog catalogs, schemas, tables, roles/groups, and RBAC/access-control lists.

• Automate platform provisioning, configuration, and deployments through Terraform, Python, SQL, Databricks Asset Bundles, and Databricks APIs.

• Implement monitoring, logging, alerting, observability, capacity planning, and performance enhancement strategies.

• Conduct Spark and SQL performance tuning, scalability testing, and troubleshooting tasks.

• Apply cost optimization strategies and contribute to DBU forecasting, financial reporting, and TCO analysis.

• Support platform upgrades, patching, versioning, release management, and the integration of new Databricks features.

• Integrate Databricks with enterprise data sources, governance tools, BI platforms, and AI/ML environments.

• Enforce security measures, encryption, networking, auditing, compliance, and high-availability/disaster-recovery controls.

• Assist with security audits, ATO activities, incident response, root-cause analysis, and operational reporting.

• Establish platform standards, reusable engineering patterns, reference architectures, runbooks, and technical documentation.

• Collaborate with architecture, security, governance, and data engineering teams.

• Provide technical guidance and mentorship to data engineers and platform users.

• Maintain monitoring dashboards, capacity planning, and KPI metrics; assess Databricks features and recommend their adoption.

⛳️ Requirements

• MA/MS degree with 12+ years of general experience in information systems and 10+ years of specialized experience.

• Experience may substitute for a degree: HS with 18+ years, AA/AS with 16+ years, BA/BS with 14+ years, or Doctorate Degree/Ph.D. with 11+ years.

• Extensive hands-on experience in implementing and administering the Databricks Lakehouse Platform, including Unity Catalog, cluster and SQL warehouse management, and job/workflow orchestration.

• Experience in performance engineering, scalability testing, and tuning of Databricks data platforms.

• Strong expertise in SQL, Spark performance tuning, and workload management.

• Proficiency in Grafana, Prometheus, Datadog, CloudWatch, and Elastic.

• Practical experience with CI/CD pipelines, including Jenkins, GitLab CI, GitHub Actions, and Azure DevOps.

• Proficiency in Python and Bash; strong SQL capabilities are essential.

• Experience operating Databricks in AWS-based or hybrid cloud environments.

• Familiarity with Docker, Kubernetes, and microservices patterns.

• Experience with telemetry, logging standards, traceability, and root-cause analysis.

• Ability to analyze large performance datasets effectively.

• Experience with version control, build/release processes, and infrastructure-as-code automation.

• Knowledge of data ingestion, ETL/ELT, lake/warehouse architectures, and distributed data processing.

• Understanding of cloud security, IAM, access control, networking, and resource governance within Databricks and Unity Catalog.

• Familiarity with federal security, compliance, and audit processes, including ATO and FedRAMP.

• Strong troubleshooting skills and the ability to optimize workload performance.

• US citizenship is not required.

• Preferred certifications include Databricks, AWS, Docker, DevSecOps, and ITIL certifications.

🏝️ Benefits

• Medical plan options, some including Health Savings Accounts.

• Dental plan options available.

• Vision plan offered.

• 401(k) plan with pre- and post-tax contributions and company matching.

• Flexible work weeks where feasible.

• Vacation, sick, and personal leave available.

• Paid holidays provided.

• Paid parental leave offered.

• Military leave available.

• Bereavement leave provided.

• Jury duty leave available.

• Up to 160 hours of paid family leave in a rolling 12-month period for eligible employees.

• Short- and long-term disability benefits included.

• Life insurance coverage provided.

• Accidental death and dismemberment insurance available.

• Personal accident insurance offered.

• Critical illness insurance provided.

• Business travel and accident insurance included.

• Remote work opportunities available.