Data Engineering Manager – GCP, AI/ML, GenAI

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee the comprehensive migration of enterprise data platforms from AWS to GCP.

• Design and implement scalable enterprise data platforms, including Data Lake and Lakehouse architectures on GCP.

• Create and deploy AI/ML, Generative AI, RAG, and MLOps solutions utilizing GCP and Vertex AI.

• Develop production-ready machine learning pipelines that encompass preparation, training, validation, deployment, monitoring, retraining, and lifecycle management.

• Design pipelines for ingestion, transformation, chunking, embedding, indexing, retrieval, as well as batch, streaming, ETL, and ELT processes.

• Evaluate AWS platforms to map AWS workloads and services to their GCP counterparts or improved services.

• Establish migration roadmaps, identify phases, dependencies, risks, rollback strategies, and target-state architectures.

• Create real-time event-driven pipelines using Pub/Sub, Dataflow/Apache Beam, BigQuery, and Cloud Storage.

• Define enterprise data models, strategies for BigQuery partitioning and clustering, and analytics consumption models.

• Implement data governance, quality, lineage, metadata, security, privacy, access control, and responsible AI standards.

• Lead Terraform infrastructure automation, CI/CD processes, deployment, testing, and environment promotion across Dev, QA, UAT, and Production.

• Enhance performance, scalability, latency, throughput, reliability, and cloud cost efficiency; set benchmarks and SLAs.

• Mentor Data Engineers, Senior Data Engineers, and Technical Leads; conduct architecture and code reviews.

• Monitor engineering progress, risks, dependencies, and delivery milestones.

• Act as the primary technical liaison for US-based stakeholders and collaborate with Business, Product, Data Science, BI, DevOps, Security, and Analytics teams.

• Convert business requirements into technical solutions while presenting architectural decisions, migration strategies, roadmaps, risks, and trade-offs.

⛳️ Requirements

• 15+ years of experience in GCP Data Engineering, Data Architecture, Cloud Engineering, AI/ML Engineering, or related technology leadership roles.

• 5+ years of substantial hands-on experience in GCP Data Engineering.

• 3+ years of extensive hands-on experience in AI/ML and GenAI.

• Proven track record in delivering AWS-to-GCP migration projects.

• Strong experience in designing enterprise Data Lake and Lakehouse platforms on GCP.

• Extensive hands-on expertise with BigQuery, Google Cloud Storage, Dataflow, Pub/Sub, Cloud Composer, Dataproc, IAM, and Terraform.

• Experience in migrating AWS data workloads, pipelines, and platforms to GCP.

• Deep understanding of AWS and GCP service mapping, migration patterns, modernization strategies, and best practices in cloud architecture.

• Experience in designing, building, and deploying AI/ML solutions on GCP using Vertex AI.

• Hands-on experience with Generative AI, LLM-based applications, RAG architectures, embeddings, vector search, prompt engineering, and enterprise AI assistants.

• Strong grasp of MLOps principles, including model training, model registry, CI/CD/CT, model deployment, monitoring, retraining, governance, and rollback strategies.

• Experience in implementing secure and responsible AI solutions, focusing on data privacy, model evaluation, access controls, auditability, and governance.

• Expert-level SQL skills along with strong proficiency in Python and PySpark.

• Strong background in data modeling, data warehousing, batch processing, and real-time data engineering.

• Experience with Terraform, Git, GitHub, Cloud Build, CI/CD pipelines, and infrastructure automation.

• Proven experience in managing and mentoring data engineering and cross-functional technical teams.

• Excellent communication skills with experience engaging with US-based stakeholders.

• Google Cloud Professional Data Engineer certification is preferred.

• Google Cloud Professional Machine Learning Engineer certification is preferred.

• Familiarity with Vertex AI Agent Builder, Vertex AI Search, Gemini models on Vertex AI, or enterprise Generative AI platforms is preferred.

• Experience with dbt, Apache Airflow, Kafka, Apache Spark, Kubernetes, Cloud Run, and API-driven architectures is preferred.

• Experience with Dataplex, Data Catalog, data lineage, metadata management, data governance, master data management, and data-quality frameworks is preferred.

• Experience supporting enterprise or regulated environments with stringent data privacy, security, compliance, audit, and governance requirements is preferred.

• Required technical skills encompass Python, PyTorch, TensorFlow, Scikit-learn, NLP, deep learning, ML algorithms, GenAI, LLMs, GPT, Gemini, Claude, Llama, prompt engineering, fine-tuning, RAG, embeddings, vector databases, semantic search, hybrid search, reranking, LangChain, LlamaIndex, LangGraph, Hugging Face, Transformers, AI agents, agentic workflows, tool/function calling, multi-agent systems, MCP, MLflow, Kubeflow, model registry, model deployment, monitoring, CI/CD, Vertex AI, Vertex AI Studio, Vertex AI Pipelines, Model Garden, Vector Search, FastAPI, Flask, REST APIs, SQL, Docker, Kubernetes, GCP, AWS, Azure, BigQuery, Dataflow, Spark, Databricks, data lakes, advanced Python, expert SQL, PySpark, Apache Spark, ETL, ELT, CDC, batch and streaming, event-driven architecture, data pipeline development and optimization, enterprise Data Lake/Lakehouse, Medallion Architecture, data warehousing, data modeling, dimensional modeling, multi-tenant modeling, schema-on-read/schema-on-write, dbt, Apache Airflow/Cloud Composer, Dataproc, Dataflow/Apache Beam, AWS Glue, Redshift, EMR, Lambda, Kinesis, Athena, CloudWatch, GCS, AWS S3, Pub/Sub, Dataplex, Data Catalog, Terraform, Git/GitHub, Cloud Build, CI/CD, Infrastructure as Code, data lineage, metadata management, data quality, monitoring, and OpenLineage.

🏝️ Benefits

• Flexible remote work options.

• Opportunity to work with global customers.

• A collaborative and innovation-driven culture.

• Continuous learning and certification opportunities.

• Chance to lead transformative AI/ML and GCP innovations as Head of Engineering.