Senior Data Engineer, GCP/dbt

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Act as a Senior Data Engineer within the collections team of a prominent financial organization.

• Develop and enhance cloud-centric data pipelines and models utilizing dbt.

• Engage in a data community of practice (chapter) and advocate for best practices.

• Assist business teams in elevating their data maturity.

• Collaborate consistently with Data Science, BI, Data Engineering, and business personnel.

• Identify issues, suggest standards, and document solutions.

• Evaluate data warehouse architecture and requirements.

• Map data, transformations, and processes across GCP services.

• Establish full-load, incremental, and CDC migration strategies.

• Create a data architecture plan on GCP.

• Design BigQuery table schemas with considerations for performance, cost, and scalability.

• Outline BigQuery partitioning and clustering strategies.

• Model Bronze, Silver, and Gold zones in Cloud Storage.

• Develop transformations using Dataproc/Spark or Dataflow to load data into BigQuery.

• Convert existing business logic and transformations into GCP.

• Execute data validation and quality assurance processes.

• Utilize Terraform to provision and manage GCP resources.

• Configure and enhance Dataproc clusters.

• Oversee networking, IAM security, and access controls in GCP.

• Optimize BigQuery queries, Spark jobs, and GCP resource utilization.

• Implement data security measures for both in transit and at rest.

• Define and enforce IAM policies while ensuring compliance with data governance standards.

• Troubleshoot performance and functional challenges affecting pipelines and GCP resources.

• Document architecture, pipelines, models, and operational procedures.

• Communicate effectively with team members, stakeholders, and other departments.

• Ensure clear communication between architecture, software components, and ongoing development activities.

⛳️ Requirements

• Minimum of 3 years of demonstrable experience with GCP.

• Minimum of 3 years of demonstrable experience with dbt.

• Minimum of 3 years of demonstrable experience with PySpark.

• Solid understanding of GitFlow.

• Extensive knowledge of BigQuery, including data modeling, query optimization, partitioning, clustering, streaming and batch loads, security, and governance.

• Experience with Cloud Storage, covering buckets, storage classes, lifecycle policies, IAM, and data security.

• Capability to provision, configure, and manage Spark/Hadoop clusters on Dataproc.

• Familiarity with Dataflow, Composer, and dbt for ELT/ETL pipelines.

• Experience in implementing Cloud IAM policies and granular access controls.

• Understanding of VPCs, networking, subnets, firewalls, and cloud security.

• Proficiency in Python and PySpark.

• Advanced SQL skills.

• Shell scripting experience.

• Knowledge of Git, GitHub, and Bitbucket.

• Familiarity with Agile methodologies, ceremonies, and proficiency using Jira.

🏝️ Benefits

• Porto Seguro health insurance, with an option to include a spouse and children.

• Porto Seguro dental insurance for employees and their dependents.

• Profit Sharing and Results (PLR).

• Childcare assistance.

• Alelo meal and food vouchers.

• Home office allowance.

• Partnerships with educational institutions, providing discounts and incentives for courses and degree programs.

• Certification incentives, including cloud certifications such as GCP, Azure, AWS, and others.

• Livelo points.

• TotalPass, offering discounted gym memberships for employees and family members.

• Mindself, with incentives for meditation and mindfulness.

• Ongoing training and professional development.