Senior Data Engineer – AWS

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Organize, gather, and process substantial volumes of data utilizing ETL tools.

• Convert the client's business goals into information management and business intelligence strategies.

• Develop and sustain data pipelines, ensuring secure access to information.

• Design and present metrics and dashboards based on collected data, guaranteeing information quality and integrity.

• Assist in the modernization of the data platform, emphasizing Lakehouse architecture, governance, quality, and preparing data for analytical and AI applications.

• Create dimensional models for use by business teams, dashboards, and reports.

• Engage with the complete data lifecycle, including ingestion, processing, storage, transformation, consumption, and governance.

• Build and refine pipelines using Python, PySpark, and AWS services.

• Collaborate with Apache Iceberg, contributing to the management of analytical tables, versioning, schema evolution, partitioning, and performance enhancements.

• Utilize query engines for large data volumes, such as Athena, Trino, Presto, or Spark SQL, working with columnar formats like Parquet and ORC.

• Execute unit and performance testing, alongside security best practices, access control, and cloud monitoring.

• Contribute to the quality, traceability, integrity, and efficiency of data processes.

⛳️ Requirements

• Experience in collecting, transforming, and loading data using ETL tools.

• Proficiency in Python and PySpark.

• Familiarity with AWS services, including Glue, EMR, Athena, SNS, Lambda, Step Functions, S3, Lake Formation, IAM, and CloudWatch.

• Understanding of NoSQL databases, such as MongoDB and DynamoDB, along with related Hadoop technologies.

• Experience with Lakehouse architecture and Apache Iceberg, including analytical table management, versioning, schema evolution, partitioning, and performance optimization.

• Proficient with query engines for handling large data volumes, such as Athena, Trino, Presto, or Spark SQL.

• Knowledge of columnar formats including Parquet and ORC.

• Experience in dimensional modeling, covering facts, dimensions, granularity, keys, hierarchies, metrics, and KPIs.

• Capability in processing structured, semi-structured, and unstructured data.

• Familiarity with conducting unit and performance testing.

• Experience in applying security best practices, access control, and monitoring in cloud settings.

• AWS Cloud Practitioner and AWS Data Engineer certifications are advantageous.

• Background in Data Quality and Data Observability is beneficial.

• Knowledge of infrastructure as code, such as Terraform, and monitoring tools like CloudWatch is a plus.

• Understanding of Data Mesh is an advantage.

• Familiarity with artificial intelligence applications in data is a bonus.

• Strong problem-solving abilities.

• Capacity to work independently and manage your own schedule.

🏝️ Benefits

• Multi-benefit card – choose how and where to utilize it.

• Scholarships for undergraduate, graduate, MBA, and language programs.

• Certification incentive programs.

• Flexible working hours.

• Competitive salary packages.

• Annual performance evaluation with a structured career development roadmap.

• Opportunities for international career advancement.

• Wellhub and TotalPass access.

• Private pension plan.

• Childcare support.

• Medical insurance.

• Dental coverage.

• Life insurance.