Senior Databricks Engineer

This is a fully remote position, open to applicants in Pennsylvania.

📋 Description

• Create and develop extensive data platforms utilizing Databricks (Delta Lake, Spark, Unity Catalog) within Azure.

• Design and sustain both batch and streaming data pipelines catering to high-volume, intricate data sources.

• Establish medallion/lakehouse architectures from scratch in greenfield settings.

• Construct and enhance data models to facilitate analytics, reporting, and downstream applications.

• Connect Databricks with enterprise systems (APIs, event streams, warehouses, ML workflows).

• Optimize Spark jobs and pipelines for performance, reliability, and cost-effectiveness at scale.

• Assist with production deployments, encompassing CI/CD pipelines, testing, and release management.

• Collaborate directly with enterprise clients to convert requirements into effective technical solutions.

• Work alongside architects, engineers, and data scientists across various workstreams.

• Maintain a balance between speed and quality, discerning when to expedite processes and when to solidify solutions.

• Make practical decisions in ambiguous and evolving situations, particularly in greenfield builds.

• Engage hands-on while concurrently directing design and methodologies across the team.

• Clearly communicate trade-offs to both technical and non-technical stakeholders.

• Operate within contemporary engineering practices (version control, code reviews, automated testing).

• Proven ability to mentor and guide data engineers and analysts.

⛳️ Requirements

• Extensive Databricks-native proficiency, encompassing experience in designing and implementing comprehensive lakehouse solutions primarily or entirely on Databricks.

• Advanced knowledge of modern Databricks architectural patterns, such as declarative pipelines / Delta Live Tables, Unity Catalog, Delta Lake, workflow orchestration, governance, performance optimization, and operational monitoring.

• Understanding of infrastructure-as-code (Terraform, Bicep), environment provisioning, and CI/CD automation (Github, Azure DevOps) for Databricks-centric platforms.

• Strong capacity for learning, technical curiosity, and comfort with AI-enabled development workflows or automation tools to expedite delivery and enhance quality.

• Familiarity with additional modern cloud data architectures and tools, including cloud-native data warehouses (Snowflake, BigQuery, Redshift), data lakes, orchestration frameworks (Airflow/Astronomer), transformation tools (dbt), catalog/governance platforms, and scalable batch or streaming data processing services (Kafka, Kinesis).

🏝️ Benefits

• Medical

• Dental

• Vision

• 401k

• Holiday pay

• Vacation

• Personal and family sick leave

• And more.