Data Engineer – Python, PySpark, AWS Glue, Amazon Athena, SQL, Apache Airflow

This is a fully remote position, open to applicants in Pakistan.

📋 Description

• Develop, enhance, and expand data pipelines and infrastructure utilizing Python, TypeScript, Apache Airflow, PySpark, AWS Glue, and Snowflake.

• Design, implement, and oversee ingestion and transformation workflows, including DAGs, alert systems, retries, SLAs, lineage, and cost management.

• Partner with platform and AI/ML teams to automate ingestion, validation, and real-time computing workflows, aiming towards a feature store.

• Incorporate pipeline health metrics into engineering dashboards to ensure comprehensive visibility and observability.

• Model data and execute efficient, scalable transformations within Snowflake and PostgreSQL.

• Create reusable frameworks and connectors to standardize the internal processes of data publishing and consumption.

⛳️ Requirements

• A minimum of 4 years of hands-on experience in production data engineering.

• Extensive practical knowledge of Apache Airflow, AWS Glue, PySpark, and Python-based data pipelines.

• Proficient SQL skills and experience in operating PostgreSQL within live environments.

• Strong grasp of cloud-native data workflows (preferably AWS) and pipeline observability (including metrics, logging, tracing, and alerting).

• Demonstrated experience in managing pipelines throughout their entire lifecycle: design, implementation, testing, deployment, monitoring, and iteration.

• Experience in performance tuning and cost optimization with Snowflake is preferred.

• Preferred experience with real-time or near-real-time processing.

• Hands-on experience with a backend TypeScript framework is a significant advantage.

• Experience with data quality frameworks, contract testing, or schema management is a plus.

• A background in developing internal developer platforms or components of data platforms is beneficial.

🏝️ Benefits

• Fully remote position.

• Compensation in USD.

• Work hours aligned with the EST time zone (9 AM to 6 PM EST) or PT time zone.