Azure Data Engineer

This is a fully remote position, open to applicants in Colombia.

📋 Description

• Create, develop, and maintain scalable and high-performance data pipelines.

• Construct ETL/ELT processes utilizing Python, PySpark, SQL, Azure Databricks, and Azure Data Factory.

• Implement integration solutions for both structured and unstructured data from various sources.

• Develop storage and processing solutions based on Data Lake and Lakehouse architecture.

• Ensure the quality, consistency, security, and availability of data.

• Optimize distributed processing workflows to handle large data volumes.

• Establish data governance, monitoring, and observability standards.

• Design and implement solutions utilizing Azure Databricks, Azure Data Factory, Azure Functions, Azure Storage Accounts, Azure Key Vault, and Azure Logic Apps.

• Automate deployment processes through CI/CD pipelines using Azure DevOps.

• Monitor and address incidents related to data processes in production settings.

• Apply best practices for security and access management in cloud environments.

• Implement secure authentication, authorization, and secret management strategies.

• Collaborate with Data Science teams to prepare, transform, and make data accessible for analytical models.

• Engage in Machine Learning, Artificial Intelligence, and NLP initiatives.

• Develop and support data pipelines for forecasting and predictive analytics models.

⛳️ Requirements

• Bachelor’s degree in systems engineering, Computer Science, Software Engineering, Data Science, or a related discipline.

• Proficiency in Python, PySpark, FastAPI, and PyTest.

• Experience with Microsoft Azure: Azure Databricks (including Databricks Notebooks and Workflows), Azure Data Factory, Azure Functions, Azure Key Vault, Azure Storage Accounts, Azure Logic Apps, and Azure DevOps.

• Familiarity with REST APIs.

• Knowledge of SQL.

• Proficient with Git.

• Desirable: Experience in Web Scraping.

• Desirable: Proficiency in Microsoft Excel for exploratory analysis and data validation.

• Understanding of data and analytics.

• Knowledge of contemporary data architectures.

• Familiarity with massive data processing and distributed computing.

• Understanding of data warehousing and data lakes.

• Knowledge of Machine Learning Engineering fundamentals.

• Familiarity with Business Intelligence and enterprise analytics.

• Understanding of forecasting and time-series analysis.

• Knowledge of credit and financial risk modeling.

• Understanding of financial analysis and performance metrics.

• Familiarity with cash flow analysis and modeling.

• Experience with Agile methodologies and Jira.

🏝️ Benefits

• Comprehensive health, dental, and vision insurance.

• Flexible work hours and remote work options.

• Opportunities for professional development and training.

• Engaging work environment that encourages innovation.