Senior Data Engineer – Azure Databricks

This is a fully remote position, open to applicants in Costa Rica.

📋 Description

• Design, construct, and sustain dependable, production-quality data pipelines utilizing Azure Databricks and Python.

• Create scalable data transformations using PySpark or Spark SQL.

• Ingest and process data from APIs, relational databases, cloud storage, and SFTP sources.

• Integrate data from ticketing, CRM, marketing, retail, and various other enterprise systems into a unified Data Hub.

• Develop and manage data warehouse architectures, dimensional models, and data marts.

• Write, optimize, and uphold SQL transformations.

• Implement data quality validations and consistency checks.

• Construct and maintain outbound data feeds for external systems.

• Investigate data discrepancies and resolve pipeline issues.

• Document data pipelines, transformation processes, dependencies, and technical workflows.

• Collaborate with team members and stakeholders to clarify requirements and identify potential risks.

• Proactively communicate progress, obstacles, and possible delays, proposing next steps to ensure deliverables remain on schedule.

⛳️ Requirements

• Over 5 years of professional experience in Data Engineering.

• Strong practical experience with Azure Databricks.

• Advanced Python expertise for data processing, transformation, and pipeline development.

• Solid experience with Apache Spark, including PySpark and/or Spark SQL.

• Advanced SQL skills, encompassing complex transformations, data validation, and query performance enhancement.

• Strong understanding of data warehousing and dimensional modeling, including facts, dimensions, and data marts.

• Experience in integrating data from REST APIs, relational databases, cloud storage, and SFTP.

• Hands-on experience with Microsoft Azure data services for data ingestion, storage, processing, and integration.

• Experience in implementing data quality checks and ensuring consistency between source systems and target data models.

• Ability to troubleshoot pipeline failures, investigate data discrepancies, and implement effective solutions.

• Strong problem-solving, analytical, and organizational capabilities.

• Ability to work independently, take ownership, and adhere to established engineering practices and documentation standards.

• Proficient English communication skills and capability to collaborate with a US-based team.

• Experience in migrating SQL-based data workloads to Azure Databricks is a plus.

🏝️ Benefits

• Option for remote work.

• Opportunity to learn quickly and collaborate with talented teams.

• Modern working practices.

• Involvement in AI-native transformation at scale.