Senior Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design and implement applications and data pipelines utilizing Python.

• Handle extensive data volumes in a distributed setting using PySpark.

• Develop and manage data transformations with dbt.

• Coordinate pipelines through Airflow.

• Utilize GitHub for version control, including pull requests, code reviews, and CI/CD integration.

• Collaborate with AWS services such as Lambda, CloudWatch, S3, Redshift, and Aurora.

• Implement and engage with Apache Iceberg.

• Employ Data Lakehouse and Medallion Architecture design patterns.

• Perform ETL/ELT processes.

• Play a key role in constructing an innovative financial platform from scratch, free from technical debt.

⛳️ Requirements

• Proven experience in Python, focusing on application development and data pipelines with an emphasis on performance, maintainability, and adhering to best practices.

• Proficient in using PySpark for the distributed processing of large datasets.

• Advanced skills in SQL.

• Familiarity with dbt for creating and managing data transformations.

• Understanding of Airflow for orchestrating pipelines.

• Substantial experience with GitHub, encompassing version control, pull requests, code reviews, and CI/CD pipeline integration.

• Knowledge of AWS services, particularly Lambda, CloudWatch, S3, Redshift, and Aurora.

• Experience with Apache Iceberg.

• Acquainted with Data Lakehouse and Medallion Architecture principles.

• Experience in executing ETL/ELT processes.

• Preferred: Experience with Databricks.

• Preferred: Familiarity with Kafka, especially in developing producers and event-driven messaging systems.

🏝️ Benefits

• Opportunities for comprehensive professional development.

• Supportive and healthy work environments.

• Engagement in real-world, impactful projects.

• An atmosphere that encourages innovation, technical ownership, and autonomy.

• The chance to create a platform from the ground up, without any technical legacy.

• Access to top-tier tools, best practices, and innovative concepts.