Senior Solutions Architect, DL Training

This is a fully remote position, open to applicants in California, +1 more state.

📋 Description

• Collaborate with Cloud Service Providers to design, construct, and showcase scalable AI training solutions utilizing NVIDIA hardware and software.

• Lead technical engagements throughout the foundation model pre-training and post-training phases, assisting customers in transitioning from architecture and proofs of concept to full production implementation.

• Create and enhance comprehensive training workflows that encompass data preparation, distributed training, checkpointing, and evaluation, including reinforcement learning pipelines.

• Develop customer-specific proofs of concept that address key technical and business challenges.

• Assist customers in diagnosing and resolving performance bottlenecks related to GPU compute, memory, communication, networking, storage, and workload orchestration.

• Collaborate with Sales Account Managers and Developer Relations Managers to identify potential opportunities and promote the adoption of NVIDIA platforms for AI training.

• Prepare and deliver technical presentations, workshops, and architectural reviews for researchers, engineers, technical leaders, and executives.

• Build strong technical relationships with customers, lead discussions on roadmaps and architecture, and work closely with NVIDIA engineering and product teams to address issues and inform future capabilities.

⛳️ Requirements

• Bachelor’s, Master’s, or PhD in Computer Science, Electrical or Computer Engineering, Statistics, Physics, or a related technical discipline, or equivalent experience.

• Over 5 years of relevant professional experience in solutions architecture, software engineering, machine learning engineering, research engineering, or similar technical roles.

• More than 3 years of practical experience with large-scale AI systems or distributed training.

• Proficient programming and debugging skills in Python.

• Hands-on experience in developing and optimizing deep learning workloads using PyTorch.

• Practical experience with both foundation model pre-training and post-training.

• Strong understanding of the overall model development lifecycle.

• Proven ownership of an end-to-end training workflow, which includes data preparation, training configuration, execution, evaluation, and troubleshooting.

• Experience with multi-GPU and multi-node training, covering distributed parallelism, mixed precision, memory optimization, and checkpointing.

• Demonstrated ability to deploy GPU workloads in cloud computing environments such as AWS, GCP, or Azure.

• Familiarity with containers and workload orchestration.

• Capacity to lead technical engagements, tackle complex problems, and effectively communicate technical concepts and trade-offs to both technical and non-technical audiences.

• Hands-on experience encompassing both pre-training and post-training of large language or multimodal models, including supervised fine-tuning and reinforcement learning.

• Experience with reinforcement learning techniques involving human feedback or verifiable rewards, including reward design, rollout generation, policy optimization, and training stability.

• Familiarity with NVIDIA NeMo Framework, Megatron Core, Megatron Bridge, PyTorch FSDP, DeepSpeed, or comparable distributed training technologies.

• System-level understanding of GPU platforms.

• Experience in profiling and optimizing training workloads.

🏝️ Benefits

• Equity

• Benefits

• Occasional travel for on-site customer engagements and industry events