Senior HPC Engineer

This is a fully remote position, open to applicants in Australia.

📋 Description

• Take responsibility for cluster performance from acceptance to production, conducting burn-in, health checks, and benchmarks such as NCCL, HPL, and storage against predefined criteria, while addressing underperformance prior to customer handover.

• Work in partnership with Deployment Solution Architects, Network Engineers, Validation teams, Customer Support, and vendors including NVIDIA and OEMs to swiftly resolve issues and incorporate production insights back into design.

• Manage the daily operations of the production HPC layer, including Slurm, GPU drivers, the CUDA stack, InfiniBand fabrics (subnet management, monitoring, firmware), and parallel storage.

• Optimize clusters for AI training and HPC workloads, focusing on NCCL configurations, GPUDirect RDMA, topology-aware scheduling, and storage performance.

• Develop monitoring solutions for GPU, fabric, and job levels, leading incident response for Xid errors and fabric degradation through to root cause analysis.

• Streamline provisioning, configuration, and validation processes using infrastructure-as-code and Git-based workflows, creating a standardized validation suite for each GPU platform and fabric type.

• Transform incident insights into automated checks, runbooks, and sustainable solutions, establishing yourself as the primary engineer for HPC and InfiniBand within the organization.

⛳️ Requirements

• Practical experience managing NVIDIA GPU infrastructure at scale, including familiarity with drivers, CUDA, DCGM, and troubleshooting GPU hardware issues.

• Demonstrated experience operating production HPC clusters with workload managers like Slurm.

• In-depth practical knowledge of InfiniBand, encompassing subnet management, RDMA, fabric monitoring, and fault diagnosis.

• Experience in benchmarking and tuning distributed workloads using tools such as NCCL tests or HPL.

• Strong Linux systems administration capabilities, including OS-level networking and storage, as well as infrastructure automation and Git-based workflows.

• Familiarity with HGX H100/H200/B200 or GB200 rack-scale systems, NVIDIA UFM, GPUDirect RDMA, and topology-aware scheduling.

• Experience with parallel file systems such as WEKA, VAST, DDN/Lustre, or GPFS, and running GPU workloads on Kubernetes or alongside OpenStack.

• Background in supporting AI/ML training, MPI-based HPC applications, research computing, or involvement in open-source HPC or infrastructure initiatives.

🏝️ Benefits

• Competitive salary and annual discretionary bonus scheme.

• Employee wellbeing benefits.

• 25 days of holiday, in addition to public holidays.

• Private Medical Insurance.

• Enhanced Pension Scheme.

• Flexible working arrangements (remote or hybrid, depending on role and location).

• Genuine ownership and autonomy — the freedom to take initiative and explore new ideas.

• Visible, impactful contributions as we scale.

• Clear career advancement opportunities in a rapidly growing company.

• A collaborative, international culture founded on trust, transparency, and ownership.

• The opportunity to help shape NexGen Cloud’s team, culture, and future.