Senior Data Engineer, Databricks Migration

This is a fully remote position, open to applicants in Ukraine.

📋 Description

• Engage in the transition of a large-scale analytical platform from BigQuery to Databricks.

• Design and implement scalable Lakehouse architectures utilizing Databricks and Delta Lake.

• Evaluate existing ETL/ELT workloads and establish migration strategies.

• Create and enhance data pipelines that handle substantial volumes of retail and analytical data.

• Apply incremental processing techniques and scalable transformation frameworks.

• Develop and sustain Spark-based data processing solutions using PySpark.

• Design and uphold the Bronze, Silver, and Gold medallion architecture layers.

• Enforce data governance and security best practices with Unity Catalog.

• Collaborate with teams in Data Science, Analytics, Product, and Customer Engineering.

• Contribute to architecture discussions and the design of technical solutions.

• Create reusable data platform components and establish engineering standards.

• Perform code reviews and enhance platform reliability and maintainability.

• Diagnose and improve complex SQL and Spark workloads.

• Assist with production deployments and platform modernization initiatives.

⛳️ Requirements

• Over 5 years of professional experience as a Data Engineer.

• Proficient programming skills in Python and advanced SQL.

• Practical commercial experience with Databricks.

• In-depth knowledge of Apache Spark, particularly PySpark.

• Experience in designing and constructing modern cloud-based data platforms.

• Background in developing ETL/ELT pipelines and large-scale data processing solutions.

• Hands-on experience with Delta Lake.

• Familiarity with Spark Declarative Pipelines.

• Experience in cluster monitoring, metrics analysis, and performance enhancement.

• Strong understanding of distributed data processing architectures.

• Solid grasp of data warehousing concepts and dimensional modeling.

• Familiarity with Airflow or similar orchestration tools.

• Experience in optimizing complex analytical SQL workloads.

• Background in implementing CI/CD practices for data engineering platforms.

• Strong troubleshooting and performance optimization capabilities.

• Ability to work collaboratively within cross-functional international teams.

• Upper-Intermediate or higher proficiency in English.

• Strong analytical and problem-solving aptitude.

• Proactive mindset with a focus on ownership.

• Capability to work independently as well as in collaboration with others.

• Effective communication and stakeholder collaboration skills.

• Passion for scalable data engineering and contemporary data platforms.

• Enthusiasm for continuous learning and technological innovation.

🏝️ Benefits

• Remote work opportunities.

• Continuous learning prospects.

• Opportunities for technological growth.

• Significant impact in engineering.

• Chance to work on complex international projects.