Senior GCP Data Engineer

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Develop and maintain data ingestion pipelines from ERP systems, production systems, quality systems, external sources, and spreadsheets.

• Manage the extraction, scheduling, and failure handling of pipelines using Apache Airflow.

• Execute the Medallion Architecture (Bronze, Silver, and Gold), storing raw data in Cloud Storage and performing data modeling in BigQuery.

• Create models for business entities and shared master data, including customers, products, and third parties.

• Enhance the semantic layer that supports dashboards.

• Ensure data quality and consistency by reconciling hub data with source systems.

• Prepare and make Gold entities accessible for AI and machine learning applications.

• Manage code versioning and oversee deployments across development, staging, and production environments.

• Monitor the execution of pipelines and address failures, aiming for operations without manual intervention.

• Maintain documentation for pipelines, data models, and technical decisions.

• Assist in client team training during the final delivery phase.

• Engage in weekly reviews and validate deliverables in collaboration with business teams.

• Collaborate in a multidisciplinary squad with architecture and data science teams to establish a centralized data hub on Google Cloud for a prominent building materials company.

⛳️ Requirements

• Over 5 years of experience as a data engineer on cloud data projects.

• More than 3 years of experience with Google Cloud, particularly BigQuery and Cloud Storage.

• At least 3 years of experience orchestrating pipelines using Apache Airflow.

• Strong SQL and analytical data modeling capabilities, with a minimum of 5 years of experience.

• Over 4 years of experience in using Python for building data pipelines.

• Familiarity with the Medallion Architecture (Bronze, Silver, and Gold).

• Proven experience in integrating data from various sources, including ERPs, APIs, files, and spreadsheets.

• Experience with code version control and managing deployments across different environments.

• Google Cloud certifications are advantageous.

• Experience in extracting data from enterprise ERP systems is a plus.

• Knowledge of tax, financial, or industrial data is beneficial.

• Familiarity with preparing data for machine learning models is a plus.

• Consulting project experience is an added advantage.

• Hands-on experience with Google Cloud (BigQuery and Cloud Storage), Apache Airflow, SQL, and Python is crucial.

🏝️ Benefits

• Continuous professional development opportunities.

• Flexibility to work remotely.

• A commitment to inclusivity, welcoming applicants with disabilities (this position is also open to candidates with disabilities).