Data Engineer

This is a fully remote position, open to applicants in India.

📋 Description

• Design, implement, and maintain ETL pipelines utilizing PySpark, Apache Airflow, and Azure Data Factory (ADF)

• Construct and enhance distributed data processing tasks using PySpark

• Orchestrate and schedule workflows through Apache Airflow

• Develop and oversee data ingestion and transformation pipelines within Azure Data Factory

• Write clear, efficient, and reusable code in Python

• Create and optimize complex SQL queries for MySQL and PostgreSQL databases

• Work with MongoDB to manage semi-structured and unstructured data

• Conduct data analysis using Pandas and NumPy to provide business insights

• Generate basic to intermediate data visualizations using Matplotlib, Power BI, and Streamlit

• Monitor data pipelines, resolve issues, and ensure data quality and performance

• Collaborate with cross-functional teams, including analysts, data scientists, and product teams

⛳️ Requirements

• 3 - 5 years of relevant experience

• Proficient in Python

• Strong expertise in MySQL and PostgreSQL

• Practical experience with MongoDB

• Experience in building ETL/ELT pipelines

• Proficient in PySpark for large-scale data processing

• Familiarity with Apache Airflow for workflow orchestration

• Experience with Azure Data Factory (ADF)

• Practical experience with Pandas and NumPy

• Capability to create visualizations utilizing Matplotlib

• Experience with Power BI for dashboards and reporting

• Exposure to Streamlit for developing data-driven applications

🏝️ Benefits

• Flexible work arrangements

• Professional development opportunities