Senior Databricks Platform Engineer

This is a fully remote position, open to applicants in Virginia.

📋 Description

• Design, configure, and manage Azure Databricks workspaces, clusters, jobs, workflows, catalogs, storage credentials, external locations, and service principals.

• Develop and enhance both batch and real-time ingestion and transformation pipelines utilizing Python, PySpark, SQL, Delta Lake, Delta Live Tables, Auto Loader, and Apache Spark.

• Implement medallion architecture, dimensional models, change data capture, schema evolution, data quality checks, retry and rollback mechanisms, and reproducible processing patterns.

• Set up Unity Catalog governance, encompassing role-based access, lineage, retention, catalog and schema provisioning, and secure integration with ADLS Gen2 and other approved storage solutions.

• Create frameworks and interfaces for governed training, inference, and reporting data across various agency initiatives.

• Automate Databricks deployments and environment configurations utilizing Terraform, Databricks Asset Bundles, GitHub Actions, Azure DevOps, or similar tools.

• Integrate Databricks with Azure Data Factory, Kafka, Airflow, APIs, Snowflake, Synapse, and other enterprise data services.

• Monitor the health of pipelines, clusters, and workspaces; diagnose performance and data issues; and perform root cause analyses.

• Assist with security, ATO, vulnerability remediation, business continuity, disaster recovery, and operational readiness activities.

• Engage in Agile planning, code reviews, demonstrations, and retrospectives.

• Document architecture, runbooks, data models, governance decisions, and deployment procedures in version-controlled repositories.

• Articulate architecture, trade-offs, risks, and operational impacts to both technical and non-technical stakeholders.

• Collaborate with data engineering, AI, cloud, security, application, and operations teams.

⛳️ Requirements

• BA/BS/MS in Computer Science, Engineering, Data Science, or equivalent experience.

• Over 5 years of experience in data engineering.

• At least 3 years of experience specifically supporting Azure Databricks and related Azure data services.

• Proficient in Apache Spark, Delta Lake, Python, SQL, and Scala.

• Experienced in designing and optimizing large-scale ETL/ELT pipelines.

• Familiar with Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, Databricks workflows, data modeling, and big data technologies.

• Skilled in managing Databricks workspaces, clusters, compute resources, security controls, access management, and cost optimization.

• Knowledgeable in CI/CD or infrastructure as code using Git, Terraform, Databricks Asset Bundles, GitHub Actions, Azure DevOps, or similar technologies.

• Experienced in Agile delivery, data governance, data security, performance troubleshooting, and stakeholder communication.

• Must meet FDIC personnel eligibility and background-investigation requirements.

• U.S. Citizenship is required.

• Must reside in the U.S., be authorized to work in the U.S., and conduct all work in the U.S.

• Preferred: experience with Azure Databricks, ADLS Gen2, Unity Catalog, Delta Live Tables, Auto Loader, Azure Data Factory, and Azure Synapse.

• Preferred: Databricks certification or equivalent demonstrated experience.

• Preferred: familiarity with streaming technologies such as Kafka, Confluent, and Structured Streaming.

• Preferred: knowledge of medallion architecture, CDC, dimensional modeling, Apache Iceberg, Snowflake, Oracle, SQL Server, or other enterprise data platforms.

• Preferred: experience with Generative AI, NLP, RAG, vector stores, agents, or Databricks AI/ML workloads.

• Preferred: familiarity with MLflow, model deployment pipelines, AWS SageMaker, AWS Bedrock, or cross-cloud architectures.

• Preferred: knowledge of BCDR, high availability, disaster recovery, data retention, and enterprise security controls.

• Preferred: experience with ATO processes, NIST or federal compliance controls, vulnerability remediation, SonarQube, Checkmarx, or Zero Trust.

• Preferred: expertise in Docker, Kubernetes, REST APIs, FastAPI, Node.js, or cloud-native application integration.

• Preferred: experience with federal data modernization programs.

🏝️ Benefits

• Reasonable accommodations for disabilities, veterans, and sincerely held religious beliefs.

• Commitment to equal opportunity employment.

• Confidential support for accommodations.

• Benefits information provided under the Transparency in Coverage Act.