Data Engineer, Mid-Level

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Design and maintain comprehensive data pipelines that encompass ingestion, processing, transformation, modeling, and data delivery.

• Construct queries and transformations utilizing advanced SQL and BigQuery.

• Create and uphold data models with Dataform or dbt.

• Establish ingestion and automation routines using Python, which includes the deployment of automated tests.

• Develop and oversee processing and orchestration workflows with Apache Airflow, preferably Cloud Composer.

• Integrate and utilize Google Cloud Platform services such as Cloud Storage, Cloud Functions, and IAM.

• Execute data quality, validation, testing, and process monitoring methodologies.

• Utilize code version control with Git and manage Pull Request workflows.

• Assist in organizing and delivering data to consumption layers and analytics tools.

• Provide support as needed for the initial configuration of the solution, encompassing user and access management.

• Maintain documentation for operational procedures and ensure runbooks are current.

• Collaborate with business units to comprehend their requirements and aid in the development of data solutions.

⛳️ Requirements

• Proven experience in Data Engineering, managing processes from ingestion to the consumption layer.

• Proficient in advanced SQL and experienced with BigQuery.

• Familiarity with data transformation and modeling using Dataform or dbt.

• Knowledge of Python for data ingestion and automation of processes.

• Experience in developing automated tests.

• Understanding of Apache Airflow, preferably with Cloud Composer.

• Proficient with Google Cloud Platform services, particularly Cloud Storage, Cloud Functions, and IAM.

• Familiarity with data quality practices and pipeline validation.

• Experience in code version control with Git and managing Pull Request workflows.

• Preferred: experience with Apache Superset.

• Preferred: understanding or experience in implementing Row-Level Security (RLS).

• Preferred: experience with access control and data segregation per client.

• Preferred: knowledge of user and access management, including SSO and MFA.

• Commitment to adhering to operational procedures and keeping runbooks updated.

• Excellent communication and collaboration abilities with business teams.

• Enthusiasm and initiative in recognizing and automating operational tasks.

🏝️ Benefits

• Contractor engagement (PJ arrangement).