Senior Databricks Engineer

This is a fully remote position, open to applicants in District of Columbia, +1 more state.

📋 Description

• Design, develop, and implement scalable data infrastructure

• Lead essential data engineering projects that support the Department of State’s “4Ps” data lake

• Collaborate with project managers, team members, and business stakeholders to gather requirements and relay progress updates

• Architect and create scalable, highly available Databricks data pipelines utilizing the Medallion Architecture

• Establish standards for data quality and develop monitoring dashboards

• Write production-ready PySpark and SQL code for intricate ETL/ELT processes

• Develop DLT and structured streaming pipelines

• Construct data ingestion pipelines through RESTful API integrations

• Implement infrastructure and pipelines using code

• Automate deployments across Development, Staging, and Production environments utilizing DABs and Azure DevOps

• Create, package, and maintain reusable custom Python libraries and modules

• Utilize data warehousing principles, including dimensional modeling and managing slowly changing dimensions

• Monitor, troubleshoot, and enhance Databricks clusters and Spark jobs

• Securely share data with external BI tools such as PowerBI and SAP BusinessObjects

• Create dashboards and functionalities directly within Databricks

• Act as a technical subject matter expert (SME), conduct code reviews, establish best practices, and provide mentorship to staff

⛳️ Requirements

• Active SECRET clearance

• Knowledge of Object-Oriented Programming (OOP) principles

• Proficiency in one or more programming or scripting languages: Java, C#, Python, JavaScript, or PowerShell

• Over 8 years of focused experience in Data Engineering

• A minimum of 3 years in a senior role concentrating on the Databricks ecosystem

• Expert-level skills in PySpark and complex SQL

• Demonstrated ability to create, package, and manage custom Python libraries, including the construction of Wheel files

• Extensive understanding of HTTP REST methods, with practical experience in extracting, paginating, and processing data from external API sources

• Profound theoretical and practical knowledge of contemporary Data Warehousing concepts

• Direct experience in constructing the Medallion Architecture

• Proven track record in automating Databricks deployments

• Practical experience in configuring and deploying Databricks Asset Bundles (DABs) using Azure DevOps, which includes creating YAML pipelines, managing service principals, and handling multi-workspace deployments

• Strong familiarity with Azure Data Lake Storage Gen2 and Azure Key Vault

• Proficient in Git-based version control workflows

• Clear and effective written and verbal communication skills for conveying technical concepts and providing customer support

🏝️ Benefits

• Medical, dental, and vision insurance

• 401(k) retirement plan

• Paid time off

• Paid parental leave

• Life and disability insurance

• Flexible spending accounts

• Commuter benefits

• Tuition reimbursement