Senior Data Engineer, Databricks Migration

This is a fully remote position, open to applicants in Portugal.

📋 Description

• Engage in the migration of a substantial analytical platform from BigQuery to Databricks.

• Design and implement scalable Lakehouse architectures utilizing Databricks and Delta Lake.

• Evaluate existing ETL / ELT workloads and outline migration strategies.

• Develop and enhance data pipelines that process large quantities of retail and analytical data.

• Apply incremental processing techniques and scalable transformation frameworks.

• Create and sustain Spark-based data processing solutions using PySpark.

• Design and maintain the Bronze, Silver, and Gold medallion architecture layers.

• Enforce data governance and security best practices through Unity Catalog.

• Collaborate with teams in Data Science, Analytics, Product, and Customer Engineering.

• Take part in architecture discussions and the design of technical solutions.

• Create reusable data platform components and establish engineering standards.

• Conduct code reviews and contribute to the reliability and maintainability of the platform.

• Diagnose and optimize complex SQL and Spark workloads.

• Assist in production deployments and activities related to platform modernization.

⛳️ Requirements

• Over 5 years of professional experience as a Data Engineer.

• Proficient programming skills in Python and advanced SQL.

• Practical commercial experience with Databricks.

• In-depth knowledge of Apache Spark, particularly PySpark.

• Experience in designing and constructing modern cloud-based data platforms.

• Proven track record in developing ETL / ELT pipelines and large-scale data processing solutions.

• Hands-on experience with Delta Lake.

• Familiarity with Spark Declarative Pipelines.

• Experience in cluster monitoring, metrics analysis, and performance enhancement.

• Strong grasp of distributed data processing architectures.

• Solid understanding of data warehousing concepts and dimensional modeling.

• Experience with Airflow or similar orchestration tools.

• Proficient in optimizing complex analytical SQL workloads.

• Experience in implementing CI / CD practices for data engineering platforms.

• Strong troubleshooting and performance optimization capabilities.

• Ability to collaborate effectively in cross-functional international teams.

• Upper-Intermediate or higher level of English proficiency.

• Familiarity with GCP cloud services, AWS, or Azure is advantageous.

• Experience in retail analytics or pricing optimization domains is a plus.

• Experience supporting machine learning or AI-related data workloads is a plus.

• Experience with platform modernization and cloud migration initiatives is a plus.

• Strong analytical and problem-solving mindset.

• Proactive and ownership-driven attitude.

• Ability to work both independently and collaboratively.

• Effective communication and stakeholder collaboration skills.

• Passion for scalable data engineering and contemporary data platforms.

• Enthusiasm for continuous learning and technological innovation.

🏝️ Benefits

• Remote work opportunities.

• Continuous learning and development opportunities.

• Opportunities for technological growth.

• Significant engineering impact.

• Involvement in complex international projects.