Senior Databricks Engineer

This is a fully remote position, open to applicants in District of Columbia.

📋 Description

• Design, develop, and deploy scalable data infrastructure while leading essential data engineering initiatives.

• Collaborate closely with project managers, team members, and business stakeholders to gather requirements, communicate progress, and ensure data aligns with customer needs.

• Capable of architecting and implementing scalable, highly available data pipelines using Databricks, strictly adhering to the Medallion Architecture (Bronze, Silver, Gold layers) to maintain data quality, consistency, and optimized reporting.

• Establish comprehensive data quality standards and create monitoring dashboards to enable business owners and responsible parties to track and proactively resolve data issues.

• Write clean, efficient, and production-ready code using PySpark and SQL for complex data extraction, transformation, and loading (ETL/ELT) processes.

• Design and construct robust data ingestion pipelines to reliably extract data from various third-party external systems via RESTful API integrations.

• Implement infrastructure and pipelines as code, utilizing Databricks Asset Bundles (DABs) integrated with Azure DevOps to manage and automate deployments across Development, Staging, and Production environments.

• Develop, package, and maintain custom Python libraries and modules to establish common frameworks utilized across multiple data projects and teams.

• Apply foundational and advanced data warehousing concepts (e.g., dimensional modeling, slowly changing dimensions) to enhance data storage and retrieval for downstream analytics and business intelligence.

• Monitor, troubleshoot, and optimize Databricks clusters and Spark jobs to ensure maximum efficiency and cost-effectiveness.

• Securely share data with external BI tools (such as PowerBI or SAP BusinessObjects) and develop dashboards and capabilities natively within the Databricks environment.

• Act as a technical subject matter expert for the data engineering team, conducting code reviews, establishing best practices, and mentoring other team members.

⛳️ Requirements

• Knowledge of Object-Oriented Programming (OOP) concepts and proficiency in one or more programming or scripting languages (Java, C#, Python, JavaScript, PowerShell).

• Experience: Over 8 years of dedicated Data Engineering experience, with at least 3 years in a senior role focused on the Databricks ecosystem.

• Core Languages: Expert-level proficiency in PySpark and complex SQL.

• Python Library Management: Demonstrated ability to create, package, and manage custom Python libraries (e.g., building Wheel files) for scalable reuse across multiple data pipelines and projects.

• API Data Ingestion: Deep understanding of HTTP REST methods (GET, POST, PUT, etc.) with practical experience extracting, paginating, and processing data from external API sources.

• Frameworks & Methodology: Profound theoretical and practical understanding of modern Data Warehousing concepts and hands-on experience building the Medallion Architecture.

• DevOps & CI/CD: Proven experience automating Databricks deployments, including hands-on experience configuring and deploying Databricks Asset Bundles (DABs) using Azure DevOps (creating YAML pipelines, managing service principals, and handling multi-workspace deployments).

• Cloud Platforms: Strong familiarity with the broader cloud ecosystem surrounding Databricks (e.g., Azure Data Lake Storage Gen2, Azure Key Vault).

• Version Control: Proficiency in Git-based version control workflows.

• Excellent written and verbal communication skills, capable of conveying technical concepts and providing customer support.

• Secret Security clearance.

🏝️ Benefits

• Health, dental, and vision insurance.

• 401K with company matching.

• Flexible spending accounts.

• Paid holidays.

• Three weeks of paid time off.

• Additional benefits.