Senior Data Engineer

This is a fully remote position, open to applicants in Spain.

📋 Description

• Design, develop, and maintain scalable data pipelines and ETL/ELT processes utilizing Azure Data Factory and Azure Databricks.

• Create data processing and transformation solutions with Python and Databricks.

• Manage large and complex datasets to ensure data quality, reliability, and performance.

• Contribute to the design and implementation of modern data architecture within the Azure ecosystem.

• Implement Medallion Architecture principles.

• Assist in the development of Data Vault and Business Vault architecture using Databricks on Azure.

• Work with Microsoft Fabric and OneLake, including the use of shortcuts for cross-platform data integration and access.

• Participate in data modeling within Azure Data Warehouse and related analytical environments.

• Collaborate with Data Architects, Developers, Analysts, and stakeholders to convert requirements into technical solutions.

• Identify opportunities for enhancements in data processing, architecture, performance, and automation.

• Ensure adherence to best practices in scalability, security, maintainability, and data governance.

⛳️ Requirements

• Proven experience in creating data pipelines using Python on Databricks.

• Familiarity with working at the curated and product layers of data engineering.

• Proficient in transforming data utilizing Data Vault 2.0, Kimball (dimensional), and 3NF modeling techniques.

• Understanding of schema evolution, slowly changing dimensions, and data harmonization techniques.

• Practical experience with DABs (Declarative Automation Bundles, formerly Databricks Asset Bundles) for deployment.

• Hands-on experience with Git repositories and CI/CD pipelines, preferably using GitHub and ADO.

• Experience in developing pipelines within an IDE environment, preferably VSCode.

• Familiarity with Microsoft Fabric and OneLake, particularly OneLake shortcuts, is a plus.

• Knowledge of data modeling in Azure Data Warehouse is advantageous.

• Experience in the financial services or wealth management sector is a bonus.

• Understanding of data governance, data quality, and contemporary cloud data architecture principles is a plus.

🏝️ Benefits

• Remote work opportunity within Spain.

• Chance to join a multicultural team and engage in international projects.

• CET working hours.