Data Engineer

This is a fully remote position, open to applicants in Mexico.

📋 Description

• Design, develop, and oversee scalable ETL/ELT pipelines for both batch and real-time data processing.

• Construct and enhance data models, Delta Tables, and Lakehouse architectures aimed at analytics and reporting.

• Create and integrate RESTful APIs and data services across various enterprise systems.

• Execute real-time and high-frequency data ingestion utilizing streaming technologies and event-driven architectures.

• Architect and manage cloud-native data solutions leveraging Azure services.

• Develop and refine Databricks Spark applications for extensive data transformation and processing.

• Ensure the quality, governance, security, and compliance of data across all platforms.

• Collaborate effectively with data scientists, analysts, application teams, and business stakeholders.

• Diagnose, monitor, and enhance the performance of pipelines and the reliability of data platforms.

• Assist in DataOps and CI/CD methodologies for the deployment and automation of data pipelines.

⛳️ Requirements

• Strong expertise in SQL and relational databases including Oracle, SQL Server, and MySQL.

• Proficient programming skills in Python, PySpark, PL/SQL, Java, or Scala.

• Practical experience with Azure Data Factory, Azure Databricks, Azure Data Lake Storage, Azure Synapse Analytics, Azure Event Hubs, Azure Functions, Azure API Management, and Azure DevOps.

• Familiarity with Databricks Lakehouse architecture, Delta Lake, and Delta Tables.

• Proficient in API development, integration, RESTful services, and microservices architecture.

• Experience in processing high-volume and high-frequency data with stringent low-latency requirements.

• Strong understanding of real-time data ingestion and streaming technologies such as Kafka, Azure Event Hubs, or Kinesis.

• Experience with Spark, Hadoop, and distributed data processing frameworks.

• Practical experience with OpenShift, Kubernetes, Docker, and containerized deployment strategies.

• Familiarity with workflow orchestration tools like Apache Airflow and Azure Data Factory.

• Understanding of data governance, security, and compliance best practices.

• Experience with Delta Live Tables, Auto Loader, and Change Data Capture.

• Knowledge of DataOps, CI/CD, and Infrastructure as Code.

• Awareness of event-driven architectures and real-time analytics platforms.

• Azure Data Engineer (DP-203) and Databricks certifications are required.

• Advanced proficiency in English.

• Essential skills: Python and Azure SQL.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Opportunities for professional development and continuous learning.

• Flexible working hours and remote work options.

• Collaborative and innovative work environment.