Senior Data Engineer – GCP

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Develop data pipelines as prioritized by the D&A team, collaborating closely with technical leadership and Product Owners.

• Design, test, and manage both batch and streaming data pipelines.

• Implement and document unit tests while assisting with UAT (user acceptance testing).

• Oversee the deployment of pipelines, including PRs, approvals, cutover, historical backfill, and scheduling.

• Convert business requirements into comprehensive technical solutions.

• Ensure compliance with Solution Architecture, Data, and Governance policies.

• Document deliverables in alignment with Project & Product standards (KT).

• Collaborate effectively with cross-functional teams and both technical and business stakeholders.

⛳️ Requirements

• Extensive experience in building pipelines on GCP (Dataflow, Pub/Sub, Cloud Storage) within production settings.

• High-level expertise in BigQuery (performance optimization, partitioning, clustering, and FinOps).

• Familiarity with Cloud Composer (Airflow) for orchestrating workflows.

• Experience in processing both structured and unstructured data (batch and streaming).

• In-depth knowledge of analytical data modeling (Star Schema, Snowflake, Medallion Architecture, SCD).

• Background in software engineering practices (Git, code review, Docker, CI/CD with Cloud Build).

• Understanding of GCP fundamentals (IAM, service accounts, VPC, security).

• Proficient in Python for data processing tasks.

• Experience with automated testing (e.g., pytest) and ensuring data quality validation.

🏝️ Benefits

• Opportunity for remote work.