Data Engineer, Mid-level

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design and improve batch and streaming data pipelines utilizing Python, Spark, and Databricks.

• Create integrations with databases, APIs, and AWS services.

• Structure and transform data in accordance with the platform's architecture and quality standards.

• Oversee production pipelines, addressing failures, performance challenges, and root cause analysis.

• Engage in code reviews and assist in the advancement of the team's engineering standards.

• Collaborate closely with business teams to comprehend requirements and convert them into data solutions.

• Uphold best practices for data quality, security, governance, and observability.

⛳️ Requirements

• Proficient in Python for data engineering, adhering to sound development practices, testing, and error management.

• Expertise in SQL, including CTEs, window functions, and query optimization techniques.

• Knowledgeable in Apache Spark / PySpark, with an understanding of partitioning, shuffling, joins, caching, and execution plans.

• Familiarity with Delta Lake, including MERGE, OPTIMIZE, VACUUM, schema evolution, and reprocessing strategies.

• Experience with Databricks, specifically in Jobs/Workflows and familiarity with Unity Catalog.

• Proficient in AWS for data, primarily S3, IAM, and integration/messaging services.

• Understanding of dimensional modeling and layered architecture, including SCD types 1 and 2.

• Experience with Git and development through pull requests.

• Proven experience in ingesting data from relational databases and REST APIs.

• Nice-to-have: experience in the financial market.

• Nice-to-have: experience with Structured Streaming.

• Nice-to-have: familiarity with Databricks Asset Bundles and Terraform.

• Nice-to-have: knowledge of CI/CD for data pipelines.

• Nice-to-have: experience with Delta Lake, including MERGE, OPTIMIZE, VACUUM, schema evolution, and reprocessing strategies.

🏝️ Benefits

• Health Insurance

• Dental Insurance

• Meal Voucher

• Food Voucher

• Life Insurance

• Total Pass

• GymPass

• Childcare Allowance

• PPR (Performance Bonus)