Principal Software Engineer – E2E Performance, Goodput

This is a fully remote position, open to applicants in California, +2 more states.

📋 Description

• Lead performance characterization initiatives in collaboration with engineering teams from major CSP and hyperscale clients, ensuring a comprehensive understanding of platform performance expectations, profiling methodologies, and tuning options tailored for specific workloads.

• Collect and analyze performance feedback from CSPs, pinpoint discrepancies between anticipated and actual throughput, and advocate for optimization priorities to NVIDIA's CUDA, NCCL, driver, and firmware teams.

• Ensure that open-source performance and stress-testing tools, including STREAM, GPU Burn, and GPU BLAST, are updated and validated for the latest NVIDIA rack-scale systems, GPU architectures, and CPU platforms.

• Partner with CSPs to guarantee that their performance and validation tools reflect the most recent GPU capabilities, changes in memory hierarchy, and platform-specific tuning parameters.

• Perform cross-CSP performance comparisons and pattern analyses to identify configuration, software, or workload variations that elucidate deployment performance discrepancies.

• Work collaboratively with CSPs to establish profiling infrastructure, benchmark harnesses, and validation configurations for deployment milestones.

• Develop test strategies and tooling requirements for performance validation related to NVIDIA's internal certification processes and customer acceptance.

⛳️ Requirements

• Over 15 years of experience in systems performance engineering, preferably in GPU/HPC/ML infrastructure.

• A Bachelor's or Master's degree in Computer Science, Computer Engineering, or a related field (or equivalent experience).

• Expertise in GPU workload profiling, including nsight systems, nsight compute, DCGM metrics, or similar instrumentation.

• Good understanding of distributed training performance dynamics, such as computation/communication overlap, pipeline bubbles, memory bandwidth utilization, and collective efficiency.

• Knowledge of statistical methods for performance analysis, including regression detection, confidence intervals, and A/B testing at scale.

• Awareness of how the entire software stack influences performance, including driver overhead, collective algorithm selection, memory allocation, scheduling, and firmware power management.

• Strong skills in data analysis and visualization (Python, pandas, dashboards).

• Ability to effectively communicate performance insights to both highly technical audiences and executive leadership.

• Proven experience in influencing multiple engineering teams to prioritize performance enhancements.

• Experience in profiling and optimizing distributed training at a scale of 1000+ GPUs (e.g., Megatron-LM, DeepSpeed, FSDP).

• Background in ML infrastructure performance within a CSP or hyperscaler environment.

• Familiarity with NVIDIA platforms (DGX, HGX, NVLink topology) and profiling tools.

• Experience in creating automated performance regression detection systems for production settings.

• Understanding of inference workload performance dynamics (e.g., vLLM, TensorRT-LLM, SGLang, continuous batching).

🏝️ Benefits

• Equity

• Benefits