Senior Data Engineer

This is a fully remote position, open to applicants in Mexico.

📋 Description

• Design, develop, and maintain scalable ETL/ELT pipelines for both batch and real-time data processing.

• Build and optimize data models, Delta Tables, and Lakehouse architectures to support analytics and reporting.

• Develop and integrate RESTful APIs and data services across various enterprise systems.

• Implement frameworks for real-time and high-frequency data ingestion utilizing streaming technologies and event-driven architectures.

• Design and manage cloud-native data solutions employing Azure Data Factory, Azure Databricks, ADLS, Event Hubs, and Synapse Analytics.

• Develop and optimize Databricks Spark applications for extensive data transformation and processing.

• Ensure data quality, governance, security, and compliance across all data platforms.

• Collaborate with data scientists, analysts, application teams, and business stakeholders.

• Troubleshoot, monitor, and enhance pipeline performance and data platform reliability.

• Support DataOps and CI/CD practices to facilitate data pipeline deployment and automation.

⛳️ Requirements

• Strong proficiency in SQL and relational databases, including Oracle, SQL Server, and MySQL.

• Strong programming skills in Python, PySpark, PL/SQL, Java, or Scala.

• Hands-on experience with Azure Cloud technologies such as Azure Data Factory, Azure Databricks, Azure Data Lake Storage, Azure Synapse Analytics, Azure Event Hubs, Azure Functions, Azure API Management, and Azure DevOps.

• Experience with Databricks Lakehouse architecture, Delta Lake, and Delta Tables.

• Expertise in API development, API integration, RESTful services, and microservices architecture.

• Experience in processing high-volume and high-frequency data with low-latency requirements.

• Strong knowledge of real-time data ingestion and streaming technologies like Kafka, Azure Event Hubs, or Kinesis.

• Experience with Spark, Hadoop, and distributed data processing frameworks.

• Hands-on experience with OpenShift, Kubernetes, Docker, and containerized deployments.

• Experience with workflow orchestration tools such as Apache Airflow and Azure Data Factory.

• Understanding of data governance, data security, and compliance best practices.

• Advanced conversational proficiency in English and Spanish is essential; language skills will be assessed.

• Candidates residing in Mexico who are foreign nationals must possess a valid INE, CURP, NSS, and RFC.

• Preferred: experience with Delta Live Tables, Auto Loader, and Change Data Capture.

• Preferred: knowledge of DataOps, CI/CD, and Infrastructure as Code.

• Preferred: familiarity with event-driven architectures and real-time analytics platforms.

• Preferred: Azure Data Engineer (DP-203) and Databricks certifications.

🏝️ Benefits

• Excellent superior benefits.