Senior Data Engineer – Databricks

This is a fully remote position, open to applicants in Poland.

📋 Description

• Take ownership of Databricks production support for the company’s data platform, including monitoring, alerting, and incident response across all production data flows.

• Maintain and report on SLA performance metrics for data pipeline delivery, ensuring transparency into platform health and accountability among both internal and external stakeholders.

• Identify and implement optimizations in pipelines that lower Databricks compute costs, enhance throughput, and minimize processing windows while tracking impacts through measurable KPIs.

• Transition legacy ETL/ELT pipelines to Databricks, creating automation tools to decrease manual intervention and guarantee uninterrupted data delivery during the transition period.

• Assist with new customer onboarding by provisioning, validating, and securing tenant data pipelines that provide reliable, isolated data from day one.

• Design and construct high-performance Databricks pipelines that ingest, transform, and serve ERP and CRM data at scale within both Azure and AWS environments.

• Manage the Delta Lake architecture, which includes schema design, partitioning strategies, data quality enforcement, and incremental processing patterns.

• Implement data security best practices across Databricks environments, addressing role-based access control, secrets management, and compliance requirements for enterprise business data.

• Establish data quality monitoring and observability across pipeline health and ML model inputs, ensuring data integrity that directly supports predictive analytics.

• Apply and maintain multi-tenant data isolation patterns, ensuring reliable and secure data delivery for enterprise customers.

• Collaborate with the Enterprise Architecture team to ensure data pipelines integrate seamlessly with the wider AI and analytics ecosystem.

• Support a globally distributed operation through an on-call rotation and after-hours incident response, adhering to SLAs across multiple time zones.

• Keep technical documentation, runbooks, and architectural decision records up to date, contributing to team knowledge sharing and operational readiness during on-call and incident response situations.

• Apply CI/CD best practices to data pipeline development, including version control, automated testing, and deployment tools to ensure consistent and reliable pipeline delivery.

⛳️ Requirements

• 4+ years of experience in data engineering.

• A minimum of 2 years of experience with Databricks or the Apache Spark ecosystem across Azure and/or AWS.

• Proficiency in PySpark, SQL, and Python, with a solid record of building and operating production-grade pipelines under SLA constraints.

• Practical experience with Delta Lake, including schema evolution, ACID transactions, lifecycle optimization/vacuuming, and both incremental and streaming processing patterns.

• Hands-on experience with performance tuning of pipelines and compute optimization in production Databricks environments.

• Strong working knowledge of PostgreSQL, including query optimization, schema design, and usage as a source or sink in production data pipelines.

• Experience with supporting and maintaining legacy ETL tools (such as SSIS, Informatica, custom Python/SQL pipelines, or similar) in production settings.

• Experience in supporting large-scale multi-tenant architectures with a focus on tenant isolation, per-tenant performance, and data privacy, including navigating tools and platforms that default to single-tenant assumptions.

• Proven ability to collaborate effectively across data science, product, and infrastructure teams, taking ownership of end-to-end delivery in a cross-functional environment.

• Strong understanding of data governance, security, and compliance principles, encompassing access control, data privacy, and the protection of sensitive enterprise data across multi-tenant environments.

• **Preferred Qualifications / Experience:**

• Experience in managing Databricks workspaces across both Azure and AWS, including cost governance, cluster management, and cross-cloud data access.

• Experience optimizing Databricks workloads within a Serverless environment, focusing on compute cost governance and performance tuning for serverless compute.

• Familiarity with Microsoft SQL Server in a data engineering or ETL context.

• Exposure to ML feature engineering or feature stores (such as Databricks Feature Store, Feast, or similar) that support predictive analytics.

• Experience with automating customer onboarding or implementing Infrastructure as Code (IaC) patterns for provisioning tenant data pipelines at scale.

• Databricks Certified Data Engineer Associate or Professional certification.

🏝️ Benefits

• Comprehensive health, dental, and vision insurance.

• Competitive salary with performance-based bonuses.

• Opportunities for professional development and continuous learning.

• Flexible work hours and remote work options.

• Collaborative and inclusive company culture.