Senior Data Engineer - AWS

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, develop, and enhance scalable data pipelines utilizing AWS services.

• Create robust processes for data ingestion, transformation, and provisioning (ETL/ELT).

• Implement and optimize solutions with AWS Glue, S3, Lambda, Athena, Redshift, EMR, Step Functions, and EventBridge.

• Design, implement, and sustain Data Lakes and contemporary data architectures in an AWS setting.

• Develop solutions in Python and SQL, emphasizing performance, scalability, and data quality.

• Handle large quantities of structured and unstructured data.

• Ensure data quality, integrity, security, and governance throughout the pipeline.

• Optimize queries, processing, and data consumption to lower costs and enhance performance.

• Collaborate with Analytics, Data Science, Software Engineering, and Architecture teams to create strategic solutions.

• Implement monitoring, observability, and error handling for data pipelines.

• Contribute to defining standards, best practices, and the advancement of the platform's data architecture.

• Assist in the analysis and resolution of critical incidents related to the data environment.

⛳️ Requirements

• Demonstrated experience as a Data Engineer in AWS environments.

• Strong understanding of core AWS services used in data architectures, including S3, Glue, Athena, Redshift, Lambda, EMR, and IAM.

• Extensive experience in developing data pipelines using ETL/ELT methodologies.

• Familiarity with Apache Spark/PySpark and Delta Lake.

• Advanced proficiency in Python and SQL.

• Experience with data modeling for analytical environments and Data Lakes.

• Knowledge of Infrastructure as Code (IaC) via AWS CloudFormation or Terraform.

• Experience with version control through Git and CI/CD practices.

• Understanding of partitioning, query optimization, and distributed processing.

• Proficiency in monitoring, observability, and troubleshooting data pipelines.

• Knowledge of cloud data architecture and best practices regarding security and data governance.

• Strong analytical capabilities, problem-solving skills, and experience in collaborative team environments.

• Preferred: experience with Databricks on AWS.

• Preferred: familiarity with Apache Airflow or other workflow orchestration tools.

• Preferred: experience with event-driven architectures.

• Preferred: knowledge of Apache Iceberg or Apache Hudi.

• Preferred: experience with Docker and Kubernetes.

• Preferred: experience in Machine Learning, AI, and MLOps projects.

• Preferred: experience in large enterprise contexts and mission-critical projects.

• Preferred: AWS certifications focused on Data Engineering, Data Analytics, or Solutions Architecture.

🏝️ Benefits

• Employment model can be either Contractor (PJ — no benefits) or CLT (Brazilian payroll with benefits).