This is a fully remote position, open to applicants in India.

• Oversee the complete migration process of around 1,000 datasets to Snowflake.

• Upgrade orchestration workflows from Apache Airflow version 2 to version 3.

• Transition legacy Hive data structures and datasets to Apache Iceberg utilizing the Snowpark Connect framework/connectors.

• Reconfigure, update, and shift Confluent/Apache Kafka streaming connections across various cloud regions.

• Modify existing data pipelines to align with Snowflake environments.

• Enhance SQL queries through the use of CTEs, window functions, and intricate joins.

• Oversee, test, and troubleshoot Directed Acyclic Graphs (DAGs) and workflow scheduling in Apache Airflow.

• Apply testing methodologies and engage in hands-on code debugging to guarantee data quality and consistency.

• Revamp streaming infrastructure across cloud regions while maintaining data quality and performance standards.

• Collaborate with the Data Platform team on a significant data platform modernization project.

• A minimum of 2+ years of practical data engineering experience with large-scale datasets, Apache Airflow, and Apache Spark.

• Strong foundation in software engineering principles.

• Practical experience in Python development.

• Advanced SQL expertise, including complex joins, CTEs, window functions, and query optimization.

• Documented experience with Snowflake data modeling and schema architecture.

• Direct experience in building and testing pipelines using Apache Airflow, Apache Kafka, and Apache Spark/PySpark.

• Proficiency with Apache Iceberg and Apache Parquet data formats.

• Familiarity with AWS S3, including data lakes, bucket partitioning, and lifecycle policies.

• Ability to create testing strategies and independently troubleshoot complex distributed data pipelines.

• Excellent communication skills and capacity to work autonomously in a fast-paced environment.

• Capability to demonstrate effective connectivity to an AWS WorkSpace region by completing the necessary connectivity test.

• Proficiency in English for conducting interviews, documentation, and daily communication.

• Availability to overlap until at least 12:00 PM Pacific Time.

• Fully remote work arrangement.

• Full-time, permanent employment.

• Client-provided AWS WorkSpace.

Data Engineer

People also viewed