Data Engineering Manager – GCP Cloud, AIML

This is a fully remote position, open to applicants in United States.

📋 Description

• Lead a comprehensive migration of data platforms from AWS to GCP.

• Design and implement scalable enterprise data platforms utilizing GCP.

• Create architectures for Data Lake and Lakehouse using Google Cloud Storage and BigQuery.

• Define the Bronze, Silver, and Gold/Atomic data layers.

• Design frameworks for batch, real-time, and event-driven data ingestion, transformation, and consumption.

• Set standards for data modeling, partitioning, clustering, storage, metadata, lineage, and data access.

• Analyze and enhance existing AWS data platforms prior to migration.

• Map AWS services and workloads to their equivalent or improved GCP services.

• Prepare migration strategies, modernization suggestions, technical roadmaps, and implementation plans.

• Architect real-time data pipelines employing Pub/Sub, Dataflow, Apache Beam, BigQuery, and Cloud Storage.

• Design and implement scalable ETL/ELT pipelines for both batch and real-time processes on GCP.

• Develop transformation frameworks utilizing Python, PySpark, SQL, Dataflow, Apache Beam, BigQuery, and dbt.

• Build orchestration workflows with Cloud Composer and Apache Airflow.

• Design enterprise data models and optimize performance and cost on BigQuery.

• Create and implement AI/ML and Generative AI solutions on GCP using Vertex AI and related GCP-native services.

• Construct production-grade machine learning pipelines for preparation, training, validation, deployment, monitoring, retraining, and lifecycle management.

• Develop solutions for RAG, enterprise search, document intelligence, AI assistants, summarization, semantic search, embeddings, vector search, and knowledge management.

• Implement MLOps practices using Vertex AI Pipelines, Model Registry, Terraform, GitHub, Cloud Build, and CI/CD pipelines.

• Establish standards for model versioning, experiment tracking, validation, testing, deployment approval, rollback, and environment promotion.

• Monitor model performance, data drift, latency, reliability, inference costs, response quality, retrieval accuracy, and GenAI risks.

• Ensure responsible AI practices, privacy, security, governance, access control, auditability, and human review processes.

• Collaborate with Data Science, Analytics, Product, BI, Security, and US-based stakeholders to deliver AI/ML and GenAI solutions.

• Serve as the main technical liaison for US-based stakeholders.

• Convert business requirements into scalable solutions for data, cloud, AI/ML, and Generative AI.

• Present decisions on architecture, migration strategies, roadmaps, implementation plans, risks, timelines, and trade-offs.

• Communicate on delivery status, dependencies, operational risks, and mitigation strategies.

• Work with business teams to establish operational, analytical, data quality, platform, and AI/ML KPIs.

⛳️ Requirements

• Over 15 years of experience in Data Engineering, Data Architecture, Cloud Engineering, AI/ML Engineering, or other technology leadership positions.

• At least 5 years of robust hands-on experience in GCP Data Engineering.

• 1–2 years of relevant experience in AIML Solutioning.

• Strong hands-on experience with AWS Data Engineering and Data Architecture.

• Proven experience in designing enterprise Data Lake and Lakehouse platforms on GCP.

• Extensive hands-on experience with BigQuery, Google Cloud Storage, Dataflow, Pub/Sub, Cloud Composer, Dataproc, IAM, and Terraform.

• Experience in migrating AWS data workloads, pipelines, and platforms to GCP.

• Deep understanding of AWS and GCP service mapping, migration patterns, modernization strategies, and cloud architecture best practices.

• Experience in designing, building, and deploying AI/ML solutions on GCP with Vertex AI.

• Hands-on experience with Generative AI, LLM-based applications, RAG architectures, embeddings, vector search, prompt engineering, and enterprise AI assistants.

🏝️ Benefits

• Medical Insurance.

• 10 company-paid holidays.

• A supportive culture with unique growth opportunities.

• Flexibility to work from home.

• Peer-to-peer recognition program.