Senior Data Engineer, Databricks Migration

This is a fully remote position, open to applicants in Ukraine.

📋 Description

• Engage in the transition of a large-scale analytical platform from BigQuery to Databricks.

• Design and implement scalable Lakehouse architectures utilizing Databricks and Delta Lake.

• Evaluate existing ETL / ELT workloads and establish migration strategies.

• Create and enhance data pipelines that process significant volumes of retail and analytical data.

• Apply incremental processing techniques and scalable transformation frameworks.

• Develop and sustain Spark-based data processing solutions employing PySpark.

• Design and manage Bronze, Silver, and Gold medallion architecture layers.

• Enforce data governance and security best practices through Unity Catalog.

• Collaborate with teams in Data Science, Analytics, Product, and Customer Engineering.

• Take part in architecture discussions and technical solution design.

• Create reusable data platform components and establish engineering standards.

• Conduct code reviews and contribute to the reliability and maintainability of the platform.

• Diagnose and enhance intricate SQL and Spark workloads.

• Assist in production deployments and activities related to platform modernization.

⛳️ Requirements

• Over 5 years of professional experience as a Data Engineer.

• Proficient programming skills in Python and advanced SQL.

• Practical commercial experience with Databricks.

• Strong knowledge of Apache Spark, with a focus on PySpark.

• Experience in designing and constructing modern cloud-based data platforms.

• Proven experience in developing ETL / ELT pipelines and large-scale data processing solutions.

• Hands-on experience with Delta Lake.

• Familiarity with Spark Declarative Pipelines.

• Experience in cluster monitoring, metrics analysis, and performance optimization.

• Solid understanding of distributed data processing architectures.

• Comprehensive knowledge of data warehousing concepts and dimensional modeling.

• Experience with Airflow or similar orchestration tools.

• Background in optimizing complex analytical SQL workloads.

• Experience in implementing CI / CD practices for data engineering platforms.

• Strong troubleshooting and performance optimization capabilities.

• Ability to collaborate effectively within cross-functional international teams.

• Upper-Intermediate English proficiency or higher.

🏝️ Benefits

• Opportunities for continuous learning.

• Potential for technological growth.

• Opportunity to make a meaningful engineering impact.

• Work on intricate international projects.