Senior Data Engineer, Databricks Migration

This is a fully remote position, open to applicants in Romania.

📋 Description

• Engage in the transition of a large-scale analytical platform from BigQuery to Databricks.

• Design and execute scalable Lakehouse architectures utilizing Databricks and Delta Lake.

• Evaluate current ETL / ELT workloads and establish migration strategies.

• Create and refine data pipelines that process substantial volumes of retail and analytical data.

• Implement incremental processing techniques and scalable transformation frameworks.

• Construct and sustain Spark-based data processing solutions using PySpark.

• Design and uphold Bronze, Silver, and Gold medallion architecture layers.

• Enforce data governance and security best practices utilizing Unity Catalog.

• Collaborate with teams across Data Science, Analytics, Product, and Customer Engineering.

• Take part in architectural discussions and the design of technical solutions.

• Develop reusable components for the data platform and establish engineering standards.

• Conduct code reviews and enhance platform reliability and maintainability.

• Diagnose and optimize complex SQL and Spark workloads.

• Assist in production deployments and the modernization of the platform.

⛳️ Requirements

• A minimum of 5 years of professional experience as a Data Engineer.

• Proficient programming skills in Python and advanced SQL.

• Practical commercial experience with Databricks.

• In-depth knowledge of Apache Spark, particularly PySpark.

• Experience in designing and constructing modern cloud-based data platforms.

• Skilled in developing ETL / ELT pipelines and large-scale data processing solutions.

• Hands-on experience with Delta Lake.

• Familiarity with Spark Declarative Pipelines.

• Experience in cluster monitoring, metrics analysis, and performance enhancement.

• Strong grasp of distributed data processing architectures.

• Solid understanding of data warehousing principles and dimensional modeling.

• Experience with Airflow or similar orchestration tools.

• Expertise in optimizing complex analytical SQL workloads.

• Experience in implementing CI / CD practices for data engineering platforms.

• Strong skills in troubleshooting and performance optimization.

• Ability to work effectively in cross-functional international teams.

• Proficient in English at an Upper-Intermediate level or higher.

🏝️ Benefits

• Employees have the flexibility to work remotely.

• Opportunities for continuous learning.

• Potential for technology growth.

• Opportunity to make a meaningful engineering impact.

• Engage in complex international projects.