Senior Databricks Engineer

This is a fully remote position, open to applicants in District of Columbia, +1 more state.

📋 Description

• Design, develop, and implement scalable data infrastructure.

• Lead essential data engineering projects utilizing the Databricks platform and contemporary data warehousing techniques.

• Construct data pipelines that support the Department of State’s “4Ps” data lake.

• Collaborate with project managers, team members, and business stakeholders to gather requirements, communicate progress, and ensure data aligns with customer needs.

• Architect and execute scalable, highly available data pipelines leveraging Databricks and the Medallion Architecture.

• Set data quality standards and implement monitoring dashboards.

• Write production-ready PySpark and SQL code for intricate ETL/ELT processes.

• Develop DLT and structured streaming pipelines.

• Design and create data ingestion pipelines using RESTful API integrations.

• Implement infrastructure and pipelines as code.

• Automate deployments across Development, Staging, and Production environments using Databricks Asset Bundles and Azure DevOps.

• Create, package, and maintain custom Python libraries and modules.

• Apply data warehousing principles, including dimensional modeling and slowly changing dimensions.

• Monitor, troubleshoot, and enhance Databricks clusters and Spark jobs.

• Securely share data with external BI tools such as PowerBI and SAP BusinessObjects.

• Develop dashboards and features directly within the Databricks environment.

• Act as a technical subject matter expert, conduct code reviews, establish best practices, and mentor team members.

⛳️ Requirements

• Active SECRET clearance.

• Understanding of Object Oriented Programming (OOP) concepts.

• Proficiency in one or more programming or scripting languages: Java, C#, Python, JavaScript, or PowerShell.

• Over 8 years of dedicated Data Engineering experience.

• A minimum of 3 years in a senior role focused on the Databricks ecosystem.

• Expert-level proficiency in PySpark and complex SQL.

• Demonstrated ability to create, package, and manage custom Python libraries, including building Wheel files.

• In-depth knowledge of HTTP REST methods such as GET, POST, and PUT.

• Practical experience in extracting, paginating, and processing data from external API sources.

• Strong theoretical and practical grasp of modern Data Warehousing concepts.

• Hands-on experience in building the Medallion Architecture.

• Proven track record in automating Databricks deployments.

• Experience configuring and deploying Databricks Asset Bundles (DABs) using Azure DevOps, including creating YAML pipelines, managing service principals, and handling multi-workspace deployments.

• Strong familiarity with Azure Data Lake Storage Gen2 and Azure Key Vault.

• Proficient in Git-based version control workflows.

• Clear and effective written and verbal communication skills for technical understanding and customer support.

• Desired: Experience with Tableau, PowerBI, Jupyter Notebooks, d3js, Elasticsearch, SOLR, and Lucene.

• Desired: Experience working with HR application (PeopleSoft) data for reporting purposes.

• Desired: Over 3 years of .NET, Java, or similar OOP experience.

• Desired: Experience writing Oracle PL/SQL queries, packages, and procedures.

• Desired: Experience in architecting Data Vault 2.0 solutions.

🏝️ Benefits

• Medical, dental, and vision insurance.

• 401(k) retirement plan.

• Paid time off.

• Paid parental leave.

• Life and disability insurance.

• Flexible spending accounts.

• Commuter benefits.

• Tuition reimbursement.