Master Data Developer

This is a fully remote position, open to applicants in Colombia.

📋 Description

• Design, construct, and sustain efficient ETL/ELT processes to ingest, transform, and deliver data within a modern Data Lake architecture.

• Develop and enhance distributed data processing workflows utilizing Python and PySpark to manage large-scale datasets effectively.

• Implement and optimize partitioning strategies for data lake storage systems (such as Delta Lake or Apache Iceberg) to achieve a balance between query performance and storage expenses.

• Craft, refine, and convert intricate SQL queries that involve CTEs, window functions, conditional expressions, and aggregations.

• Transition and modernize data pipelines from traditional RDBMS platforms to cloud-native analytics solutions.

• Operate confidently with AWS-native services, including Glue, Athena, and Redshift.

• Collaborate with infrastructure and DevOps teams to provision and manage data resources using Infrastructure as Code (IaC) tools like CloudFormation, CDK, or Terraform.

• Monitor the health and performance of data pipelines using CloudWatch and other observability tools.

⛳️ Requirements

• Strong experience in ETL processes and data pipeline development using AWS.

• High proficiency in Python as the primary programming language, with proven experience in writing and optimizing PySpark code for distributed data processing.

• Comprehensive understanding of SQL, including complex queries (CTEs, window functions, aggregations, conditional expressions) and experience translating workloads from legacy RDBMS platforms.

• Practical experience with AWS Glue (Jobs, Catalog, Triggers, Workflows), Athena, and Redshift.

• In-depth knowledge of Data Lake architectures and partitioning strategies to enhance performance and reduce costs.

• Solid grasp of object-oriented programming (OOP) principles and experience in working with reusable code libraries.

• Comfortable using Git, Shell scripts, and operating in Linux environments.

• Familiarity with observability, monitoring, and metric tracking practices.

• Advanced/Fluent English proficiency.

🏝️ Benefits

• Premium Healthcare.

• Meal voucher.

• Maternity and Parental leaves.

• Mobile services subsidy.

• Sick pay-Life insurance.

• CI&T University.

• Colombian Holidays.

• Paid Vacations.