Senior Data Engineer – GCP, DBT

This is a fully remote position, open to applicants in Brazil.

📋 Description

• **Analysis and Planning of Loads/Pipelines:**

• Assess the architecture and requirements of the data warehouse.

• Align data, transformations, and processes with GCP services (Cloud Storage, BigQuery, Dataproc).

• Establish a data migration strategy (full load, incremental, CDC).

• Formulate a data architecture plan on GCP.

• **Design and Data Modeling on GCP:**

• Develop table schemas in BigQuery, focusing on performance, cost, and scalability.

• Specify partitioning and clustering strategies for BigQuery.

• Structure data zones in Cloud Storage (Bronze, Silver, and Gold).

• **ELT/ETL Pipeline Development:**

• Construct data transformation routines utilizing Dataproc (Spark) or Dataflow to populate data into BigQuery.

• Convert business logic and existing transformations into GCP.

• Establish data validation and quality control mechanisms.

• **Provisioning and Infrastructure Management:**

• Employ IaC tools (Terraform) for the provisioning and management of GCP resources (BigQuery datasets/tables, Cloud Storage buckets, Dataproc clusters).

• Configure and enhance Dataproc clusters for various workloads.

• Oversee networking, security (IAM), and access management on GCP.

• **Performance and Cost Optimization:**

• Enhance queries in BigQuery to minimize costs and boost performance.

• Adjust and optimize Spark jobs on Dataproc.

• Monitor and refine GCP resource utilization to manage costs.

• **Data Security and Governance:**

• Implement and ensure data security both in transit and at rest.

• Define and enforce IAM policies to regulate access to data and resources.

• Ensure adherence to data governance policies.

• **Monitoring and Support:**

• Resolve performance and functionality issues related to data pipelines and GCP resources.

• **Documentation:**

• Document the architecture, data pipelines, data models, and operational procedures.

• **Communication:**

• Effectively communicate with team members, stakeholders, and other business areas.

• Ensure clear communication between architecture definitions and software components, supporting the evolution and quality of the team’s projects.

• **Jira / Agile Methodologies:**

• Familiarity with agile methodologies, ceremonies, and proficiency in the Jira tool.

⛳️ Requirements

• **Google Cloud Platform (GCP):**

• **BigQuery:** Extensive expertise in data modeling, query optimization, partitioning, clustering, data loading (both streaming and batch), security, and data governance.

• **Cloud Storage:** Experience in managing buckets, storage classes, lifecycle policies, access control (IAM), and data security.

• **Dataproc:** Proficient in provisioning, configuring, and managing Spark/Hadoop clusters, job optimization, and integrating with other GCP services.

• **Dataflow/Composer/DBT:** Familiarity with orchestration and data-processing tools for ELT/ETL pipelines.

• A proven track record of at least 3 years of experience in GCP.

• A minimum of 3 years of experience in DBT (preferred).

• At least 3 years of experience in PySpark.

• Proven experience with GitFlow.

• **Cloud IAM (Identity and Access Management):** Experience in implementing security policies and fine-grained access control.

• **VPC, Networking and Security:** Knowledge of networks, subnets, firewall rules, and cloud security best practices.

• **Programming Languages:**

• **Python and PySpark:** Crucial for automation scripts, pipeline development, and integration with GCP APIs.

• **SQL (advanced):** Required for BigQuery, DBT, and data transformations.

• **Shell Scripting:** Necessary for task automation.

• **Version Control:**

• Git / GitHub / Bitbucket.

🏝️ Benefits

• 🏥 Porto Seguro Health Plan

• 🦷 Porto Seguro Dental Plan

• 💰 Profit Sharing (PLR)

• 👶 Childcare Assistance

• 🍽️ Alelo Meal and Food Vouchers

• 💻 Home Office Allowance

• 📚 Partnerships with Educational Institutions

• 🚀 Support for Certifications, including Cloud

• 🎁 Livelo Points

• 🏋️‍♂️ TotalPass

• 🧘‍♂️ Mindself