Senior Data Engineer – Databricks, PySpark, SQL

This is a fully remote position, open to applicants in Portugal, +1 more country.

📋 Description

• Create, develop, and sustain scalable data ingestion and transformation pipelines on a Databricks-based Lakehouse platform.

• Establish Bronze, Silver, and Gold data layers.

• Integrate data from enterprise systems, APIs, and external sources.

• Develop processing workflows utilizing PySpark and SQL.

• Implement reusable engineering frameworks.

• Optimize pipelines for scalability and cost-effectiveness.

• Convert established architectural standards into actionable engineering solutions.

• Support initiatives related to data governance, lineage, cataloging, and access management.

• Enable BI and Analytics teams to access reliable, business-ready data.

• Engage in Agile ceremonies.

• Contribute to project delivery using Git-based version control and CI/CD practices.

• Work collaboratively with Data Architects, Analytics teams, and business stakeholders.

⛳️ Requirements

• Minimum of 4 years of professional Data Engineering experience.

• At least 2 years of practical Databricks development experience.

• Strong proficiency in Databricks and PySpark for developing and maintaining data processing pipelines.

• Solid SQL development and optimization skills.

• Experience with Azure Data Platform services.

• Demonstrated experience in designing and implementing ETL/ELT pipelines, encompassing data ingestion and transformation.

• Understanding of Lakehouse architecture and Medallion Architecture, including Bronze, Silver, and Gold data layers.

• Experience in integrating data from various enterprise systems, APIs, and external sources.

• Knowledge of data quality, pipeline reliability, performance optimization, and data security principles.

• Familiarity with using Git and CI/CD practices in data engineering workflows.

• Ability to work effectively with Data Architects, BI and Analytics teams, and business stakeholders.

• Professional working proficiency in English.

• Availability to work remotely from Portugal or Spain.

• Preferred: Experience with Delta Live Tables (DLT).

• Preferred: Familiarity with Unity Catalog and Databricks data governance capabilities.

• Preferred: Experience with Microsoft Fabric.

• Preferred: Previous participation in Data Hub or Enterprise Data Platform programs.

• Preferred: Experience in developing reusable data engineering frameworks and components.

• Preferred: Knowledge of data lineage, cataloging, and enterprise data governance practices.

• Preferred: Experience in optimizing cloud data workloads for performance and cost efficiency.

🏝️ Benefits

• 15th month salary.

• Health insurance that covers your family.

• Birthday off.

• Mobility program for digital nomads.

• Genuine work-life balance.