Senior Databricks Platform Engineer

This is a fully remote position, open to applicants in Virginia.

📋 Description

• Design, configure, and manage Azure Databricks workspaces, clusters, jobs, workflows, catalogs, storage credentials, external locations, and service principals.

• Create and enhance batch and real-time ingestion and transformation pipelines.

• Implement medallion architecture, dimensional models, change data capture, schema evolution, data quality checks, retry and rollback mechanisms, and reproducible processing patterns.

• Set up Unity Catalog governance and secure integrations with ADLS Gen2 and authorized storage.

• Develop frameworks and interfaces for AI and analytics teams to access governed training, inference, and reporting data.

• Automate Databricks deployments and environment configurations using infrastructure-as-code and CI/CD tools.

• Integrate Databricks with Azure Data Factory, Kafka, Airflow, APIs, Snowflake, Synapse, and other enterprise data services.

• Monitor the health of pipelines, clusters, and workspaces; diagnose performance and data issues; and assist in root cause analysis.

• Support activities related to security, ATO, vulnerability remediation, business continuity, disaster recovery, and operational readiness.

• Engage in Agile planning, code reviews, demonstrations, and retrospectives.

• Document architecture, runbooks, data models, governance decisions, and deployment procedures in version-controlled repositories.

• Communicate architecture, trade-offs, risks, and operational impacts to both technical and non-technical stakeholders.

• Collaborate with data engineering, AI, cloud, security, application, and operations teams.

⛳️ Requirements

• BA/BS/MS in Computer Science, Engineering, Data Science, or equivalent experience.

• Over 5 years of experience in data engineering.

• At least 3 years specifically supporting Azure Databricks and related Azure data services.

• Proficiency in Apache Spark, Delta Lake, Python, SQL, and Scala.

• Practical experience in designing and optimizing large-scale ETL/ELT pipelines.

• Familiarity with Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Databricks workflows, data modeling, and big data technologies.

• Experience in configuring and managing Databricks workspaces, clusters, compute resources, security controls, access management, and cost optimization.

• Background in implementing CI/CD or infrastructure as code using Git and tools like Terraform, Databricks Asset Bundles, GitHub Actions, Azure DevOps, or similar technologies.

• Experience with Agile delivery, data governance, data security, performance troubleshooting, and stakeholder communication.

• Ability to meet FDIC personnel eligibility and background-investigation requirements.

• U.S. Citizenship is required due to federal contract stipulations.

• Candidate must reside in the U.S., be authorized to work in the U.S., and carry out all work within the U.S.

• Preferred: Experience with Databricks, Azure, Unity Catalog, streaming, Generative AI, MLflow, BCDR, federal compliance, Docker, Kubernetes, REST APIs, and federal data modernization.

• Databricks certifications or comparable demonstrated experience are preferred.

🏝️ Benefits

• Equal opportunity employer.

• Reasonable accommodations for disabled veterans, individuals with disabilities, and individuals with sincerely held religious beliefs.

• Confidential accommodation support.

• Benefit offerings referenced under the Transparency in (Benefits) Coverage Act.