Senior Data Engineer – Databricks Migration

This is a fully remote position, open to applicants in Poland.

📋 Description

• Engage in the migration of a large-scale analytical platform from BigQuery to Databricks.

• Design and implement scalable Lakehouse architectures utilizing Databricks and Delta Lake.

• Analyze ETL/ELT workloads and establish migration strategies.

• Develop and enhance high-volume retail and analytical data pipelines.

• Implement incremental processing strategies and scalable transformation frameworks.

• Construct and maintain Spark-based data processing solutions using PySpark.

• Design and uphold Bronze, Silver, and Gold medallion architecture layers.

• Enforce data governance and security best practices through Unity Catalog.

• Collaborate with Data Science, Analytics, Product, and Customer Engineering teams.

• Take part in architecture discussions and the design of technical solutions.

• Create reusable data platform components and establish engineering standards.

• Conduct code reviews and contribute to the reliability and maintainability of the platform.

• Troubleshoot and enhance complex SQL and Spark workloads.

• Support production deployments and initiatives for platform modernization.

⛳️ Requirements

• Over 5 years of professional experience as a Data Engineer.

• Proficient programming skills in Python.

• Advanced SQL proficiency.

• Practical experience with Databricks.

• In-depth knowledge of Apache Spark, predominantly PySpark.

• Experience in designing and building contemporary cloud-based data platforms.

• Proven track record in developing ETL/ELT pipelines and large-scale data processing solutions.

• Hands-on experience with Delta Lake.

• Familiarity with Spark Declarative Pipelines.

• Experience in cluster monitoring, metrics analysis, and performance optimization.

• Strong grasp of distributed data processing architectures.

• Solid understanding of data warehousing principles and dimensional modeling.

• Experience with Airflow or similar orchestration tools.

• Proven ability to optimize complex analytical SQL workloads.

• Familiarity with implementing CI/CD practices for data engineering platforms.

• Strong troubleshooting capabilities and performance optimization skills.

• Ability to work collaboratively in cross-functional international teams.

• Upper-Intermediate English proficiency or higher.

• Experience with GCP cloud services, AWS, or Azure is advantageous.

• Background in retail analytics or pricing optimization is a plus.

• Experience supporting machine learning or AI-related data workloads is an asset.

• Familiarity with platform modernization and cloud migration initiatives is beneficial.

🏝️ Benefits

• Opportunities for ongoing learning.

• Technology growth prospects.

• Significant engineering impact.

• Engagement in complex international projects.