Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design, construct, and enhance the architecture of the company’s data platform.

• Develop and sustain ETL/ELT pipelines, encompassing API extraction, transformations, incremental loads, and Change Data Capture (CDC).

• Create data models for Data Warehouses, Data Lakes, and Data Marts using dimensional modeling techniques (Star Schema, Snowflake Schema).

• Implement both batch and streaming processing utilizing event-driven architecture and asynchronous processing.

• Ensure the performance and reliability of databases through relational modeling, indexing, partitioning, query optimization, and replication.

• Establish practices for Data Quality, Data Lineage, Data Catalog, and Data Governance.

• Orchestrate pipelines while managing historical data with versioning and traceability.

• Implement observability for pipelines, including logs, metrics, monitoring, and alerts.

• Provide consistent and near real-time data for dashboards, operational analytics, and business intelligence.

• Lay the data groundwork for prospective Machine Learning initiatives.

• Automate processes and deployments using GitHub Actions and Docker.

• Collaborate with product, engineering, and business teams to outline requirements and KPIs.

⛳️ Requirements

• Proficiency in languages: Python and advanced SQL.

• Experience with databases: PostgreSQL and MySQL, including relational modeling, indexing, partitioning, query optimization, replication, and performance tuning.

• Expertise in Data Modeling: Data Warehouse, Data Lake, Star Schema, Snowflake Schema, Data Mart, dimensional modeling, normalization, and denormalization.

• Knowledge of ETL / ELT processes: constructing pipelines, API extraction, data transformation, incremental loading, and Change Data Capture (CDC).

• Familiarity with Data Processing: batch processing, streaming, asynchronous processing, and Event-Driven Architecture.

• Experience with AWS services: S3, Lambda, SQS, RDS, CloudWatch, and IAM.

• Proficient in messaging services, specifically Amazon SQS.

• Version control experience with Git and GitHub.

• Understanding of observability practices: logs, metrics, pipeline monitoring, and alerts.

• Automation skills using GitHub Actions.

• Experience with containerization technologies like Docker.

• Knowledge in Data Engineering fields: Data Quality, Data Lineage, Data Catalog, Data Governance, data versioning, pipeline orchestration, and historical data management.

🏝️ Benefits

• Flexible working hours

• Career progression plan