Engineering Manager – AWS to GCP Data Migration, AI/ML, GenAI

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee the complete migration of enterprise data platforms from AWS to GCP.

• Evaluate AWS architecture, data pipelines, workloads, dependencies, and operational workflows.

• Establish target-state GCP architecture, migration roadmaps, phases, dependencies, risks, and rollback plans.

• Facilitate architecture reviews and technical design discussions.

• Design and implement scalable enterprise GCP Data Lake and Lakehouse platforms.

• Create frameworks for data ingestion, transformation, consumption, batch, and real-time ETL/ELT processes.

• Architect streaming pipelines utilizing Pub/Sub, Dataflow/Apache Beam, BigQuery, and Cloud Storage.

• Develop enterprise data models, including BigQuery partitioning, clustering, and analytics consumption frameworks.

• Set up data governance, quality, lineage, metadata, ownership, security, IAM, encryption, and access policies.

• Spearhead Terraform infrastructure automation, CI/CD, testing, deployment, and environment standards.

• Enhance performance, SLAs, and costs across BigQuery, Dataflow, Spark, Cloud Storage, and streaming workloads.

• Mentor Data Engineers, Senior Data Engineers, and Technical Leads while establishing engineering standards and priorities.

• Monitor progress, risks, dependencies, milestones, coding, testing, security, and documentation practices.

• Act as the primary technical liaison for US-based stakeholders and collaborate with Business, Product, Data Science, BI, DevOps, Security, and Analytics teams.

• Design and implement AI/ML and Generative AI solutions on GCP utilizing Vertex AI and related services.

• Construct production ML pipelines for preparation, training, validation, evaluation, deployment, monitoring, retraining, and lifecycle management.

• Develop RAG, enterprise search, document intelligence, AI assistants, summarization, semantic search, embeddings, vector search, and knowledge-management solutions.

• Implement MLOps, including model versioning, experiment tracking, validation, testing, deployment approvals, rollback, and environment promotion.

• Monitor model performance, data drift, latency, reliability, inference cost, response quality, retrieval accuracy, hallucination, and prompt-injection risks.

• Ensure responsible AI practices, privacy, security, governance, access control, auditability, and human review.

• Collaborate with stakeholders to identify, prioritize, and deliver high-value AI/ML and GenAI use cases.

⛳️ Requirements

• Over 15 years of experience in Data Engineering, Data Architecture, Cloud Engineering, AI/ML Engineering, or similar technology leadership roles.

• At least 5 years of substantial hands-on experience in GCP Data Engineering.

• Strong practical experience with AWS Data Engineering and Data Architecture.

• Demonstrated expertise in delivering AWS-to-GCP migration projects.

• Significant experience in designing enterprise Data Lake and Lakehouse platforms on GCP.

• Proficient hands-on experience with BigQuery, Google Cloud Storage, Dataflow, Pub/Sub, Cloud Composer, Dataproc, IAM, and Terraform.

• Experience in migrating AWS data workloads, pipelines, and platforms to GCP.

• Thorough understanding of AWS and GCP service mapping, migration patterns, modernization strategies, and cloud architecture best practices.

• Experience in designing, building, and deploying AI/ML solutions on GCP using Vertex AI.

• Hands-on experience with Generative AI, LLM-based applications, RAG architectures, embeddings, vector search, prompt engineering, and enterprise AI assistants.

• Strong understanding of MLOps, including model training, model registry, CI/CD/CT, model deployment, monitoring, retraining, governance, and rollback strategies.

• Proven experience in implementing secure and responsible AI solutions, including data privacy, model evaluation, access controls, auditability, and governance.

• Expert-level SQL skills alongside strong Python and PySpark capabilities.

• Extensive experience in data modeling, data warehousing, batch processing, and real-time data engineering.

• Familiarity with Terraform, Git, GitHub, Cloud Build, CI/CD pipelines, and infrastructure automation.

• Experience in managing and mentoring data engineering and cross-functional technical teams.

• Excellent communication skills with a track record of working with US-based stakeholders.

• Google Cloud Professional Data Engineer certification is preferred.

• Google Cloud Professional Machine Learning Engineer certification is preferred.

• Familiarity with Vertex AI Agent Builder, Vertex AI Search, Gemini models on Vertex AI, or enterprise Generative AI platforms is preferred.

• Experience with dbt, Apache Airflow, Kafka, Apache Spark, Kubernetes, Cloud Run, and API-driven architectures is preferred.

• Experience with Dataplex, Data Catalog, data lineage, metadata management, data governance, master data management, and data-quality frameworks is preferred.

• Experience supporting enterprise or regulated environments with rigorous data privacy, security, compliance, audit, and governance requirements is preferred.

• Required proficiency in AWS services including Amazon S3, AWS Glue, AWS Glue Data Quality, Amazon Redshift/Redshift Serverless, Amazon Athena, AWS Step Functions, AWS DMS, AWS Lake Formation, and IAM.

• Required proficiency in GCP services including BigQuery, Google Cloud Storage, Pub/Sub, Dataflow/Apache Beam, Cloud Composer/Airflow, Dataproc/Spark, Cloud Monitoring, Cloud Logging, Dataplex/Data Catalog.

• Required knowledge of ETL/ELT, CDC, batch and streaming data processing, event-driven architecture, data pipeline optimization, enterprise Data Lake/Lakehouse, Medallion Architecture, data modeling, dimensional modeling, multi-tenant data modeling, schema-on-read/schema-on-write, data lineage, metadata management, data governance, dbt, Apache Airflow, Terraform, Git/GitHub, Cloud Build, CI/CD, data quality frameworks, and OpenLineage (a plus).

🏝️ Benefits

• Flexible remote work environment.

• Exposure to global enterprise customers.

• Collaborative, innovation-driven engineering culture.

• Continuous learning and certification opportunities.

• Opportunity to lead large-scale AWS-to-GCP cloud transformation initiatives.

• Engage in enterprise Data Lakehouse and analytics modernization projects.