AWS Data Engineer

This is a fully remote position, open to applicants in Mexico.

📋 Description

• Create reusable Python packages for both internal and public use

• Streamline ETL processes and schedule jobs utilizing Airflow DAGs

• Monitor pipeline executions and reprocess records with errors

• Coordinate pipelines to execute either sequentially or concurrently

• Diagnose and resolve issues within data pipelines

• Manage data imports and exports in CSV, JSON, XML, and other formats, preferably leveraging S3 or other cloud storage solutions

• Enhance complex SQL queries for data warehouse technologies, ideally Snowflake

• Execute unit, integration, and system tests in Python

• Conduct DBT tests for data validation and quality assurance

• Generate code from configurations utilizing Python and Jinja templates

• Establish GitHub version control and CI/CD workflows

• Oversee AWS data storage and security measures, including S3, ECS, and IAM

• Integrate AWS Lambda, Apache Airflow, DBT models, EMR, S3, Redshift, and Snowflake

• Develop API integrations with Salesforce, SFTP, SharePoint, OneDrive, and other platforms

• Design intricate DBT models, macros, snapshots, and associated documentation

• Supervise pipelines and set up alerts for workflow failures through CloudWatch and DataDog

• Function as an individual contributor while also collaborating within a team

• Engage directly with clients

⛳️ Requirements

• Over 10 years of professional experience

• Exceptional communication and client-facing abilities

• Strong attention to detail

• Capability to work both independently and collaboratively as part of a team

• Fully hands-on experience with the required skills

• Advanced expertise in Python, encompassing code structures, modules, packages, classes, inheritance, multithreading, and functional programming

• Proven experience in developing reusable Python packages

• Background in automating ETL processes and scheduling jobs using Airflow DAGs

• Advanced SQL proficiency, including complex joins, CTEs, and subqueries

• Experience in optimizing SQL queries for data warehouse technologies, preferably Snowflake

• Proficiency in unit, integration, and system testing in Python

• Expertise in implementing DBT tests for data validation and quality assurance

• Experience in generating code using Python and Jinja templates

• Familiarity with GitHub and setting up CI/CD processes from the ground up

• Comprehensive understanding of AWS S3, ECS, and IAM, focusing on organization and security best practices

• Knowledge of AWS security best practices, IAM roles, encryption standards, secure coding guidelines, and DBT profile access configurations

• Experience with an AI IDE tool

• Nice to have: Experience with AWS Lambda

• Nice to have: Proficiency in Apache Airflow on AWS

• Experience in integrating Airflow with AWS services and DBT models

• Experience with API integrations for Salesforce, SFTP, SharePoint, OneDrive, and other systems

• Nice to have: Proficiency in DBT Core/Cloud, including complex models, incremental models, snapshots, documentation, and reusable macros

• Nice to have: Familiarity with AWS CloudWatch and DataDog for monitoring and alerts in pipelines

🏝️ Benefits

• Competitive salary and performance-based bonuses

• Flexible working hours and remote work options

• Ongoing training and professional development opportunities

• Health, dental, and retirement benefits

• Collaborative and inclusive company culture