ML Platform Engineer, Distributed Training

This is a fully remote position, open to applicants in United States.

📋 Description

• Collaborate with the client's infrastructure team on the GKE training platform, focusing on GPU node pools, job queueing and scheduling, quota management, and capacity planning.

• Identify and resolve platform failures by addressing root causes.

• Optimize multi-node GPU networking, including RDMA/GPUDirect and NCCL, to enhance training throughput.

• Execute, scale, and troubleshoot multi-node distributed training using PyTorch and Ray Train.

• Analyze jobs and eliminate bottlenecks across computing, memory, input pipelines, and interconnects.

• Design and implement performance benchmarks for the cluster and training stack.

• Develop and sustain model evaluation harnesses.

• Integrate telemetry and monitoring into training and the platform.

• Conduct end-to-end pipeline tests against actual user workloads.

• Actively identify issues, address gaps, propose solutions, communicate status and risks, and support scientists utilizing the platform.

• Articulate and justify design decisions to client engineers and scientists.

• Adapt readily to shifting client priorities and team structures.

• Develop document intelligence and retrieval-augmented generation (RAG) features from prototype to scalable production services.

• Convert client business requirements into high-performance AI architectures.

⛳️ Requirements

• Over 5 years of engineering experience, with recent hands-on involvement in GPU training infrastructure and large-scale training.

• Proficient in Python and shell scripting.

• Extensive knowledge of Google Gemini, GPT-4, or LLaMA.

• In-depth understanding of prompt engineering and fine-tuning.

• Expertise in vector databases, including Vertex AI Vector Search and pgvector.

• Experience in implementing semantic search.

• Practical experience with Google Cloud (Vertex AI) and scalable software architectures.

• Proficient in LangChain, LlamaIndex, or similar orchestration frameworks.

• Strong Python software engineering skills, including production-quality code, testing, and code review.

• Familiarity with Kubernetes on Google Cloud, GPU node pools, job queueing, and capacity/quota management.

• Personal experience in running and debugging multi-node distributed training jobs with PyTorch and Ray Train.

• Knowledge of GPUDirect/RDMA, NCCL tuning, and multi-node GB200 or H200 topology.

• Experience in designing performance benchmarks and conducting model evaluations.

• Ability to explain system operations and rationales without depending on AI coding tools.

• Proven track record of ownership, effective communication, and accountability in a high-pressure team environment.

• Experience in healthcare or life sciences data and familiarity with HIPAA regulations are advantageous.

• Knowledge of observability and telemetry for training platforms is a plus.

🏝️ Benefits

• Comprehensive Health Insurance

• Paid Leave (Vacation/PTO)

• Paid Holidays

• Sick Leave

• Parental Leave

• Bereavement Leave

• 401(k) Employer Match

• Employee Referral Bonuses