Senior Data and Agent Platform Engineer, GCP

This is a fully remote position, open to applicants in Brazil.

📋 Description

• Develop and maintain batch and streaming data ingestion, transformation, and quality pipelines utilizing BigQuery, Dataflow, Dataproc, Pub/Sub, and Cloud Storage.

• Structure and organize the Data Lake/Lakehouse within BigQuery and BigLake, focusing on layers, partitioning, and optimizing cost and performance.

• Establish data contracts, detailing schemas, freshness, and SLAs, while monitoring source quality and lineage.

• Prepare data for AI applications by cleaning, anonymizing, chunking, and generating embeddings for Vertex AI RAG Engine, Vertex AI Search, and vector stores.

• Ensure compliance with LGPD when managing sensitive health data by implementing masking, anonymization, policy tags, and data residency controls.

• Design data access across projects using Cloud IAM, Dataplex, BigLake, and Analytics Hub, adhering to the principle of least privilege and maintaining audit trails.

• Expose data and internal systems as agent tools via MCP servers and MCP Toolbox, facilitating integration with Vertex AI Agent Engine and ADK.

• Manage MLOps/LLMOps pipelines using Vertex AI Pipelines, including evaluation, implementing guardrails with Model Armor, and monitoring drift and response quality.

• Provision data and AI infrastructure through code with Terraform and CI/CD practices.

• Monitor costs by business area, project, and model using billing exports to BigQuery and labels.

• Implement comprehensive observability with OpenTelemetry, Cloud Logging, Cloud Monitoring, and Cloud Trace.

• Assist in developing resilience and disaster recovery strategies.

⛳️ Requirements

• Proficient in Python and SQL.

• Extensive experience in GCP data engineering, specifically with BigQuery, Cloud Storage, Dataflow or Dataproc, and Pub/Sub.

• Experience in orchestrating pipelines using Cloud Composer/Airflow, Cloud Workflows, or similar tools.

• Understanding of data governance and security protocols in GCP, including Cloud IAM, Dataplex, policy tags, and Cloud KMS.

• Familiarity with Terraform and CI/CD processes, including Cloud Build, GitHub Actions, or comparable tools.

• Practical experience in multi-project GCP environments with organizations and folders.

• Knowledge of distributed and event-driven systems, involving queues, retries, idempotency, and consistency.

• Experience preparing data for at least one production LLM application using RAG or tool-enabled agents.

• Awareness of LGPD as it pertains to sensitive data, including anonymization and masking practices.

• Nice to have: Familiarity with Vertex AI, Gemini, Model Armor, agent frameworks, MCP servers, vector stores, GraphRAG, HL7/FHIR standards, Cloud Healthcare API, Data Mesh, hub-and-spoke architecture, and GCP FinOps.

• Certifications such as Google Cloud Professional Data Engineer, Professional Machine Learning Engineer, or FinOps Certified Practitioner are advantageous.

• Strong communication skills to effectively collaborate with data scientists, AI teams, business stakeholders, and external partners.

• A robust security mindset and ethical judgment when dealing with sensitive patient data.

• Ability to clearly convey technical trade-offs.

• Commitment to documenting and assigning ownership of pipelines, data sources, and tools across teams.

🏝️ Benefits

• Porto Seguro medical insurance, with the option to include a spouse and children.

• Porto Seguro dental coverage for employees and their dependents.

• Participation in the Profit Sharing and Results Program (PLR).

• Assistance with childcare expenses.

• Alelo food and meal vouchers.

• Home office allowance.

• Collaborations with educational institutions, providing discounts and incentives for courses and degree programs.

• Support for certifications, including cloud certifications in GCP, Azure, AWS, and other platforms.

• Accumulation of Livelo points.

• TotalPass offering discounted gym memberships for employees and family members.

• Mindself program promoting meditation and mindfulness.

• Continuous professional development opportunities.

• A collaborative work environment.