Data Engineer – Python, PySpark, AWS Glue, Amazon Athena, SQL, Apache Airflow

This is a fully remote position, open to applicants in Pakistan.

📋 Description

• Develop, enhance, and scale data pipelines and infrastructure utilizing Python, TypeScript, Apache Airflow, PySpark, AWS Glue, and Snowflake.

• Create, operationalize, and oversee data ingestion and transformation workflows, including DAGs, alert mechanisms, retries, SLAs, lineage, and cost management.

• Partner with platform and AI/ML teams to automate data ingestion, validation, and real-time computing workflows; aim towards establishing a feature store.

• Incorporate pipeline health and metrics into engineering dashboards to ensure comprehensive visibility and observability.

• Model data and execute efficient, scalable transformations within Snowflake and PostgreSQL.

• Develop reusable frameworks and connectors to standardize the internal processes of data publishing and consumption.

⛳️ Requirements

• Over 4 years of hands-on experience in production data engineering.

• Extensive, practical knowledge of Apache Airflow, AWS Glue, PySpark, and Python-based data pipelines.

• Strong SQL proficiency and experience managing PostgreSQL in active environments.

• Comprehensive understanding of cloud-native data workflows (preferably AWS) and pipeline observability (including metrics, logging, tracing, and alerting).

• Demonstrated experience managing data pipelines from start to finish: design, implementation, testing, deployment, monitoring, and iteration.

🏝️ Benefits

• Fully remote position.

• Compensation will be in USD.

• Work hours are synchronized with the EST time zone (9 AM to 6 PM EST) or PT time zone.