Senior Data Engineer – Azure Databricks

This is a fully remote position, open to applicants in Costa Rica.

📋 Description

• Design, construct, and sustain production-level data pipelines utilizing Azure Databricks and Python.

• Create scalable data transformations employing PySpark or Spark SQL.

• Ingest and process data from APIs, relational databases, cloud storage, and SFTP sources.

• Integrate data from ticketing, CRM, marketing, retail, and various enterprise platforms into a unified Data Hub.

• Develop and uphold data warehouse structures, dimensional models, and data marts.

• Write, optimize, and manage SQL transformations.

• Implement data quality validations and consistency checks.

• Construct and maintain outbound data feeds for third-party systems.

• Investigate data discrepancies and resolve pipeline issues.

• Document data pipelines, transformation logic, dependencies, and technical processes.

• Collaborate with team members and stakeholders to clarify requirements, identify risks, and deliver dependable data solutions.

• Communicate progress, obstacles, and potential delays proactively while proposing next steps.

⛳️ Requirements

• Over 5 years of professional experience in Data Engineering.

• Extensive hands-on experience with Azure Databricks.

• Proficient in Python for data processing, transformation, and pipeline development.

• Solid experience with Apache Spark, including PySpark and/or Spark SQL.

• Advanced SQL skills, covering complex transformations, data validation, and query performance optimization.

• Strong understanding of data warehousing and dimensional modeling, including facts, dimensions, and data marts.

• Experience in integrating data from REST APIs, relational databases, cloud storage, and SFTP.

• Practical experience with Microsoft Azure data services.

• Experience in implementing data quality checks and validating consistency between source systems and target data models.

• Ability to troubleshoot pipeline failures and investigate data discrepancies.

• Capable of working independently, taking ownership, and adhering to engineering practices and documentation standards.

• Strong English communication skills.

• Ability to collaborate effectively with a US-based team.

• Experience in migrating SQL-based data workloads to Azure Databricks is a plus.

🏝️ Benefits

• Employees have the flexibility to work remotely.

• Opportunity for rapid learning and taking ownership of projects.

• Collaboration with skilled teams.

• Investment in modern working methodologies.

• Involvement in large-scale AI-native transformation initiatives.