GPU Fabric Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Validate and troubleshoot InfiniBand and RoCE fabrics during the setup and expansion of GPU clusters.

• Adjust fabric performance settings to optimize distributed AI workloads (NCCL, MPI, collective operations).

• Oversee and manage fabrics utilizing NVIDIA UFM (Unified Fabric Manager) for monitoring health, topology, and performance visibility.

• Identify and resolve fabric-level challenges, including link errors, congestion, packet loss, and path asymmetry.

• Enhance RDMA transport configurations, PFC/ECN behavior, and lossless queue arrangements for GPU traffic.

• Assess fabric performance benchmarks and guarantee line-rate throughput for AI tasks.

• Collaborate with GPU Engineers to link fabric health with workload performance outcomes.

• Work with networking teams on fabric provisioning, configuration, and troubleshooting.

• Address fabric alerts and degradation incidents across operational GPU clusters.

• Document procedures for fabric troubleshooting, tuning parameters, and validation runbooks.

⛳️ Requirements

• 3–7 years of experience in network engineering, HPC fabric, or GPU infrastructure.

• Practical experience with InfiniBand and/or RoCE fabrics in GPU cluster settings.

• Familiarity with NVIDIA UFM for fabric management, monitoring, and diagnostics.

• Solid understanding of RDMA transport, lossless Ethernet design, and congestion management (PFC, ECN, DCQCN).

• Experience in GPU cluster networking and distributed communication libraries (NCCL, MPI).

• Knowledge of GPU platforms and their interconnect requirements (NVIDIA NVLink, NVSwitch, ConnectX).

• Experience with fabric diagnostic tools (ibstat, ibqueryerrors, perfquery, etc.).

• Proficient in Python or Bash for scripting and validation tasks.

• Basic knowledge of Linux systems and server hardware.

• Strong troubleshooting and analytical capabilities across network and system layers.

🏝️ Benefits

• 100% company-covered insurance premiums for employee medical, dental, and vision plans.

• 401(k) plan with a 100% match up to 4%, alongside immediate vesting.

• Annual Professional Development Reimbursement of $2,500.

• 11 Holidays plus Paid Time Off Accrual and a Rollover Plan.

• Increased PTO at 3-year and 10-year anniversaries, along with a 1-month paid sabbatical every 5 years and an Anniversary Bonus each year.

• $500 stipend for remote office setup in the first year, followed by $400 each subsequent year.

• Internet reimbursement of up to $75 per month.

• Gym membership reimbursement of up to $50 per month.

• Company-sponsored Wellable subscription.