Senior Data Engineer – Databricks Expert

This is a fully remote position, open to applicants in Poland.

📋 Description

• Design, develop, and maintain data pipelines within Databricks utilizing Python, PySpark, and SQL.

• Construct and uphold Delta Lake tables and transformation layers adhering to bronze, silver, and gold architecture.

• Integrate data from various source systems into consistent, analytics-ready datasets.

• Execute incremental processing, change data capture, and schema evolution as necessary.

• Develop and manage production workflows, encompassing dependencies, scheduling, retries, and monitoring.

• Enhance Spark workloads, SQL queries, and compute usage to boost performance and control costs.

• Establish automated data quality checks, validations, and pipeline testing.

• Assist with data access management, governance, and lineage utilizing Unity Catalog.

• Maintain reusable code, technical documentation, and Git-based development workflows.

• Collaborate with business stakeholders, data engineers, analysts, and machine learning experts to convert requirements into dependable data solutions.

• Contribute to the complete delivery lifecycle, from client discussions and business requirements analysis to solution design, implementation, and production deployment.

⛳️ Requirements

• Strong commercial experience with Databricks, encompassing the development and operation of production data pipelines.

• Advanced proficiency in Python, PySpark, and SQL.

• Hands-on experience with Apache Spark, distributed data processing, and performance troubleshooting.

• Practical knowledge of Delta Lake, including incremental loads, merge operations, and schema management.

• Solid understanding of lakehouse architecture, ETL/ELT patterns, and data modeling.

• Experience orchestrating and monitoring workflows in Databricks.

• Familiarity with Unity Catalog, including permissions, data organization, and lineage.

• Experience in optimizing pipeline performance and compute resource utilization.

• Knowledge of cloud storage and services in at least one major cloud environment: Azure, AWS, or GCP.

• Experience with Git, code reviews, automated testing, and CI/CD workflows.

• Strong analytical and communication skills, capable of working directly with international stakeholders.

• Good command of the English language.

🏝️ Benefits

• Opportunities to engage with modern data engineering and machine learning technologies.

• An annual self-development budget.

• The chance to contribute to a diverse range of intriguing projects.

• Internal workshops and knowledge-sharing sessions.

• Support for personal branding through articles, conference presentations, and leading internal workshops.

• Flexible working hours.

• The possibility of remote work.

• A chillout room, complimentary beverages, and team and company events.

• A friendly work environment.

• MultiSport membership.

• LuxMed healthcare coverage.