Senior Data Developer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, construct, and sustain robust ETL/ELT processes to ingest, transform, and distribute data within a modern Data Lake architecture.

• Develop and enhance distributed data processing workflows utilizing Python and PySpark to efficiently manage large-scale datasets.

• Implement and optimize partitioning strategies for data lake storage frameworks (like Delta Lake or Apache Iceberg) to achieve a balance between query performance and storage expenses.

• Write, optimize, and convert complex SQL queries that include CTEs, window functions, conditional expressions, and aggregations.

• Migrate and modernize data pipelines from outdated RDBMS platforms to cloud-native analytics environments.

• Employ object-oriented programming principles to aid in the development of in-house libraries, promoting code reusability and standardization.

• Work proficiently with AWS-native services including Glue (Jobs, Catalog, Triggers, Workflows), Athena, Redshift, S3, Lambda, EventBridge, and related data services.

• Collaborate with infrastructure and DevOps teams to provision and manage data resources utilizing Infrastructure as Code (IaC) tools such as CloudFormation, CDK, or Terraform.

• Monitor the health and performance of data pipelines using CloudWatch and other observability tools, proactively addressing issues and enhancing reliability.

• Ensure data integrity, consistency, and compliance throughout pipelines and storage layers.

• Implement metric tracking and observability frameworks to ensure transparency in data workflows and SLAs.

• Support data catalog management and metadata governance practices.

• Partner with data analysts, scientists, and business stakeholders to comprehend requirements and translate them into scalable technical solutions.

• Contribute to technical documentation, participate in code reviews, and share knowledge within the team.

• Remain updated with emerging data engineering practices, tools, and cloud-native innovations.

⛳️ Requirements

• Significant experience in ETL processes and data pipeline development using AWS.

• Strong expertise in Python as the primary programming language, with proven experience in writing and optimizing PySpark code for distributed data processing.

• Comprehensive understanding of SQL, including complex queries (CTEs, window functions, aggregations, conditional expressions) and experience in migrating workloads from legacy RDBMS platforms.

• Practical experience with AWS Glue (Jobs, Catalog, Triggers, Workflows), Athena, and Redshift.

• Solid understanding of Data Lake architectures and partitioning strategies to enhance performance and reduce costs.

• Good grasp of object-oriented programming (OOP) principles and experience in working with reusable code libraries.

• Comfortable operating in Git, Shell scripts, and Linux environments.

• Familiarity with observability, monitoring, and metric tracking practices.

• Advanced/Fluent English proficiency.

• Nice to Have: Experience with open table formats such as Delta Lake or Apache Iceberg.

• Knowledge of TypeScript.

• Practical experience with Infrastructure as Code (IaC) tools such as CloudFormation (preferred), CDK, or Terraform.

• Familiarity with additional AWS services like SageMaker AI, ECS, RDS, DynamoDB, IAM, or EventBridge.

• Experience with Pandas for data manipulation and analysis.

• Exposure to machine learning workflows or AI-driven data initiatives.

🏝️ Benefits

• Health and dental insurance.

• Meal and food allowance.

• Childcare assistance.

• Extended paternity leave.

• Partnership with gyms and health and wellness professionals via Wellhub (Gympass) TotalPass.

• Profit Sharing and Results Participation (PLR).

• Life insurance.

• Continuous learning platform (CI&T University).

• Discount club.

• Free online platform dedicated to physical, mental, and overall well-being.

• Pregnancy and responsible parenting course.

• Partnerships with online learning platforms.

• Language learning platform.

• And many more!