Senior Staff Engineer – Infrastructure Systems

This is a fully remote position, open to applicants in California.

📋 Description

• Take ownership of the GPU software foundation, including kernel, NVIDIA driver, firmware, CUDA runtime and libraries, and the container toolkit.

• Validate driver and CUDA releases across various GPU generations, frameworks, and model families.

• Maintain compatibility matrices while identifying issues related to correctness, performance, and memory regressions.

• Diagnose GPU failures, including Xid errors, CUDA errors, cuBLAS failures, out-of-memory problems, fragmentation issues, and GPUs disconnecting.

• Develop GPU acceptance, burn-in, diagnostics, memory, interconnect, and real-workload testing protocols.

• Enhance GPU health visibility and implement self-healing capabilities using telemetry from DCGM, drivers, kernel logs, and BMC.

• Automate GPU fault classification, node draining, resets, and replacement processes.

• Manage GPU workload exposure through container runtime, device plugins, GPU Operator, MIG, sharing policies, and tenant isolation.

• Collaborate with hardware and node operations teams to bring up new GPU generations and pods.

• Validate drivers, firmware, NVLink, and NVSwitch, while automating future bring-up activities.

• Benchmark image, video, audio, and LLM inference across different GPU generations and software stacks.

• Contribute performance results to inform placement, pricing, and hardware procurement decisions.

• Partner with NVIDIA and upstream projects to address reproducible driver and firmware bugs and apply open-source fixes.

• Leverage AI agents for log and Xid triage, test generation, release qualification, and runbook automation.

• Work collaboratively with node operations, Managed Kubernetes, inference performance, and Sonic Inference Engine teams.

⛳️ Requirements

• 12+ years of experience in building and operating systems software, with significant time dedicated to GPU software in production at scale.

• Extensive hands-on expertise with the NVIDIA software stack: kernel driver and modules, CUDA runtime and libraries, NVML, DCGM, Container Toolkit, and version interactions.

• Proven experience in diagnosing GPU failures across hardware, firmware, driver, runtime, and application layers.

• Familiarity with Xid errors, ECC and row remapping, PCIe and NVLink errors, thermal faults, and memory faults.

• Strong understanding of Linux internals: kernel, device drivers and modules, PCIe, NUMA, IOMMU, cgroups, containers, perf, eBPF, and ftrace.

• Regular coding experience.

• Proficient in Python and at least one systems programming language: C, C++, Rust, or Go.

• Comfortable using CUDA for writing tests or reproducing bugs.

• Experience managing GPUs in a Kubernetes environment and on bare metal, including device plugins, GPU Operator, MIG, image builds, and large-scale provisioning.

• Working knowledge of GPU inference frameworks such as PyTorch, TensorRT, and ONNX Runtime.

• Understanding of memory allocation, compilation and caches, and multi-GPU execution.

• Ability to produce clear technical writing for design documents, postmortems, and ticket updates.

• Daily application of AI tools to enhance productivity.

• Comfortable taking ownership of results in a fast-paced, ambiguous environment.

• Nice to have: Experience with GPU driver, CUDA library, or GPU system software at a GPU vendor, cloud provider, or HPC center.

• Nice to have: Familiarity with Blackwell hardware, NVLink/NVSwitch topologies, and liquid-cooled racks.

• Nice to have: Knowledge of NCCL, GPUDirect, RDMA over RoCE or InfiniBand.

• Nice to have: Expertise in firmware qualification across BIOS, BMC, NIC, and GPU.

• Nice to have: Understanding of inference serving internals, including TensorRT-LLM, vLLM, SGLang, or custom diffusion/video pipelines.

• Nice to have: Experience with AMD or custom accelerator software stacks.

• Nice to have: Participation in an open-source project used in a production environment.

• Nice to have: Knowledge of multi-tenant isolation and confidential computing on shared GPU hardware.

• Nice to have: Experience managing a 24x7 service with established service levels.

🏝️ Benefits

• Remote-first work arrangement.

• Biannual in-person company retreats.

• Core collaboration hours complemented by flexible hours outside of them.

• Flexible scheduling options.

• Downtime provided after major release pushes.

• Generous paid time off policy, including vacation, sick days, and public holidays.

• Meaningful stock options.

• Paid maternity, paternity, and caregiver leave.