Senior Data Engineer – AWS

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Oversee schemas, backups, read replicas, and parameter groups while crafting efficient SQL queries.

• Formulate queries for both SQL and NoSQL databases, choosing the suitable paradigm for each task.

• Design Glue Jobs, set up crawlers, and execute Spark Jobs on EMR.

• Construct Spark/PySpark transformations to read from and write to various data sources.

• Generate and query Iceberg/Delta tables utilizing time travel and snapshots.

• Enhance Athena queries via partitioning, columnar formats, and workgroups.

• Manage data and permissions within Lake Formation.

• Establish schemas, load data using COPY, create analytical queries, and manage users in Redshift.

• Develop DAGs in Airflow, configure connections, and schedule workflows.

• Execute lineage, cataloging, data classification, access controls, schema validation, and versioning.

• Set up ElastiCache clusters, implement cache-aside patterns, and manage TTLs.

⛳️ Requirements

• Senior-level expertise in Data and AWS.

• Understanding of Redis and Memcached, along with caching concepts and usage criteria.

• Proficient in relational databases, RDS, and both SQL and NoSQL paradigms.

• Familiarity with ETL processes, AWS Glue, EMR, and distributed processing using Apache Spark.

• Experience with ACID compliance in Data Lakes, open formats, Athena, Lake Formation, and data governance.

• Knowledge of MPP Data Warehousing and Redshift.

• Acquainted with Airflow, DAGs, data lineage, traceability, security, compliance, Lakehouse architecture, and Data Contracts.

• Preferred: background in cache architecture and optimization, connection pooling, Multi-AZ failover, performance tuning, IAM, and encryption.

• Preferred: understanding of access-pattern-oriented modeling, consistency strategies, and query optimization.

• Preferred: advanced knowledge of Glue, EMR, Spark, Data Quality, schema evolution, and formats like Iceberg, Delta, and Hudi.

• Preferred: experience with Athena, Glue Catalog, Lake Formation, granular permissions, and cross-account sharing.

• Preferred: in-depth knowledge of Redshift, materialized views, workload management, and data sharing.

• Preferred: experience in developing intricate DAGs, MWAA, custom operators, retries, and observability.

• Preferred: proficiency in data governance, lineage, metadata, quality metrics, Data Contracts, and schema registries.

🏝️ Benefits

• Multi-benefit card – choose how and where to utilize it.

• Tuition assistance for undergraduate, graduate, MBA, and language courses.

• Incentives for certification programs.

• Flexible working hours.

• Competitive salary packages.

• Annual performance evaluations with a defined career development plan.

• Opportunities for international careers.

• Access to Wellhub and TotalPass.

• Private pension plan.

• Childcare support.

• Medical insurance coverage.

• Dental insurance coverage.

• Life insurance coverage.