Senior Data Engineer, Databricks, PySpark

This is a fully remote position, open to applicants in Poland.

📋 Description

• Design and create scalable data products utilizing Databricks, Delta Lake, and PySpark.

• Construct and uphold data pipelines that process intricate datasets from ERP, procurement, sustainability, and external sources.

• Enhance workloads for improved performance, scalability, and cost efficiency.

• Develop reusable engineering patterns.

• Establish automated data quality controls across the entire data lifecycle.

• Detect and resolve data issues prior to their impact on reporting.

• Implement CI/CD pipelines, automated deployments, testing frameworks, and Infrastructure as Code.

• Support platform security measures and access management.

• Collaborate with sustainability experts, business analysts, and reporting teams to gather requirements, design solutions, and manage production releases.

⛳️ Requirements

• 5–7 years of experience in designing, developing, and managing data platforms and pipelines.

• Extensive practical experience with Databricks, including Azure Databricks and Databricks Workflows, in live environments.

• Expert-level knowledge in PySpark, Python, Spark SQL, Data Modelling, and Data Pipeline Design.

• Experience in implementing CI/CD pipelines for data engineering tasks.

• Proficient in Git-based development and version control.

• Strong understanding of data lineage, metadata management, governance, auditing, and validation frameworks.

• Proficient in English, both written and verbal.

• Proactive, self-motivated individual with robust troubleshooting and root-cause analysis capabilities.

• Familiarity with sustainability reporting, ESG data, and regulatory reporting requirements (preferred).

• Knowledge of procurement, supplier, finance, and ERP data domains (preferred).

• Experience in developing Databricks applications, dashboards, or user-facing data tools (preferred).

🏝️ Benefits

• Fully remote work opportunity from Warsaw.

• Full-time schedule of 40 hours per week.

• B2B cooperation agreement.