Senior AI Infrastructure Systems Engineer – R&D, GPU, AI

This is a fully remote position, open to applicants in United States.

📋 Description

• Formulate technical hypotheses and design specific tests.

• Analyze system logs along with low-level hardware data.

• Differentiate between GPU, host, firmware, interconnect, thermal, and physical hardware failures.

• Diagnose and resolve intricate hardware and platform failures within high-performance AI infrastructure.

• Examine issues related to Linux, GPU servers, PCIe, firmware, power, and cooling.

• Identify true root causes for problems that lack existing troubleshooting procedures.

• Transform ambiguous issues into evidence-based conclusions and replicable troubleshooting methods.

• Benchmark systems for performance, stability, power efficiency, thermal behavior, and workload characteristics.

• Conduct thorough problem investigations, root cause analyses, and performance optimizations.

⛳️ Requirements

• At least 5 years of practical systems engineering experience in Linux, server hardware, firmware, PCIe, and GPU or high-performance computing platforms.

• Extensive experience with Linux for hardware debugging, system-level troubleshooting, and platform analysis.

• Familiarity with the Linux kernel and experience in kernel-level debugging or troubleshooting.

• Strong understanding of modern server architecture, especially in high-performance, GPU-based settings.

• Solid knowledge of NVIDIA GPU platforms and diagnostic tools, including nvidia-smi, XID and SXID analysis, GSP and driver behavior, NVLink, NVSwitch, Fabric Manager, DCGM diagnostics, and NCCL testing.

• Comprehensive understanding of PCIe, covering topology, enumeration, root complexes, endpoints, switches, bridges, retimers, link speed and width, AER and DPC errors, completion timeouts, and link failures.

• Insight into the connections between protocol-level errors and physical-layer or signal-integrity issues.

• Experience in benchmarking systems for performance, stability, power efficiency, thermal behavior, and workload characteristics.

• Profound understanding of firmware interactions across BIOS/UEFI, BMC, CPLD/FPGA, GPU firmware, NIC firmware, and other platform components.

• Experience with low-level hardware communication and debugging using I²C, SMBus, PMBus, register maps, bit masks, byte- and word-level data, and device datasheets.

• Proven ability to troubleshoot complex hardware, software, and networking issues.

• Experience with in-depth problem investigation, root cause analysis, and performance optimization in cloud or high-performance computing environments.

• Strong analytical and problem-solving skills with a performance-oriented mindset.

• Proficiency in scripting and automation using Python or Go.

• Applicants must be authorized to work in the country of application and provide proof of employment eligibility as a condition of hire.

🏝️ Benefits

• 100% company-paid medical, dental, and vision coverage for employees and their families.

• 401(k) plan with up to 4% company match and immediate vesting.

• 20 weeks of paid parental leave for primary caregivers.

• 12 weeks of paid parental leave for secondary caregivers.

• Remote work reimbursement of up to $85/month for mobile and internet expenses.

• Company-funded short-term, long-term, and life insurance coverage.

• Competitive salary.

• Opportunities for career advancement and learning.

• Flexibility and ownership in work.

• A collaborative and innovative workplace culture.

• The chance to contribute to impactful AI projects.

• An international environment with talented teams.