Databricks Administrator

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee and manage Databricks workspaces across development, testing, and production environments.

• Set up and administer clusters, pools, jobs, and workflows for ETL, streaming, and machine-learning tasks.

• Monitor the health, capacity, and performance of the platform, addressing issues proactively.

• Handle Databricks runtime versions, libraries, cloud-storage, and data-service integrations.

• Execute backup, recovery, and business-continuity measures.

• Configure and manage IAM roles, groups, and permissions.

• Implement security best practices, including encryption, network controls, and data-access policies.

• Collaborate with security and compliance teams to meet organizational and regulatory requirements.

• Support data governance standards, including Unity Catalog, catalogs, schemas, tables, and permissions.

• Optimize cluster settings, autoscaling, and job scheduling for enhanced performance and cost efficiency.

• Analyze usage metrics and costs, implementing strategies for cost control and chargeback/showback.

• Suggest and execute platform-tuning strategies for handling large-scale data workloads.

• Collaborate with data engineers, analysts, and data scientists to understand workload needs and recommend Databricks best practices.

• Provide Tier 2/3 support for incidents and service requests related to Databricks.

• Create and maintain documentation, runbooks, and standard operating procedures for the platform.

• Conduct user onboarding, training, and knowledge-sharing sessions.

• Automate workspace and cluster provisioning, configuration, and policy enforcement using Infrastructure as Code.

• Integrate Databricks with CI/CD pipelines for code deployment and environment management.

• Support best practices in version control for notebooks, jobs, and configuration.

⛳️ Requirements

• A minimum of 8 years of experience managing cloud-based data platforms or big data environments.

• Hands-on experience in administering Databricks.

• A Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field, or four additional years of relevant work experience in lieu of a degree.

• Strong knowledge of Databricks workspaces, clusters, jobs, pools, and notebooks.

• Familiarity with cloud storage solutions such as Azure Data Lake Storage, S3, or GCS.

• Understanding of networking concepts.

• Experience with identity and access management within the relevant cloud provider.

• Proficiency in Python, PowerShell, or Bash scripting and automation.

• Familiarity with Spark concepts and performance considerations.

• Experience in monitoring and resolving platform performance and reliability issues.

• Strong analytical and problem-solving abilities.

• Capacity to convey complex technical concepts to both technical and non-technical audiences.

• Customer-oriented mindset with a focus on reliability and quality of service.

• Ability to work both independently and collaboratively in a fast-paced, agile environment.

• Must obtain and maintain a Public Trust clearance.

• Must provide onsite fingerprinting at a designated facility.

• U.S. residency for the past five years, subject to specified absence limits.

• Travel required is less than 10%.

• Preferred: Databricks certifications.

• Preferred: Experience with Unity Catalog and fine-grained data access controls.

• Preferred: Familiarity with CI/CD tools such as Azure DevOps, GitHub Actions, or Jenkins.

• Preferred: Knowledge of Infrastructure as Code, especially Terraform.

• Preferred: Experience with log aggregation and monitoring tools.

• Preferred: Understanding of Delta Lake, structured streaming, and data Lakehouse architectures.

• Preferred: Background in enterprise security and compliance.

🏝️ Benefits

• Comprehensive medical plan options, including those with Health Savings Accounts.

• Options for dental plans.

• Vision plan availability.

• 401(k) plan featuring pre-tax and post-tax contributions with company matching.

• Flexible work weeks where feasible.

• Paid time off for vacation, sick leave, personal days, holidays, parental leave, military leave, bereavement, and jury duty.

• Generally, 15 days of paid leave each calendar year.

• An additional 10 paid holidays annually.

• Up to 160 hours of paid family leave in a rolling 12-month period for eligible employees.

• Short- and long-term disability benefits.

• Life insurance coverage.

• Accidental death and dismemberment insurance.

• Personal accident insurance.

• Critical illness insurance.

• Business travel and accident insurance.

• AI-driven career tool identifying career progression and learning opportunities.

• Internal mobility support team.

• Wellness package offerings.

• Competitive salary.

• Military-friendly workplace.

• Identity verification and security processes in place.