AI Performance Engineer

This is a fully remote position, open to applicants in California.

📋 Description

• Assess and enhance the overall performance of distributed AI workloads, including training and multi-node inference, across CPU/GPU clusters and networking fabrics such as Omni-Path, Ethernet, and InfiniBand.

• Benchmark, characterize, and fine-tune open-source and industry-standard workloads and benchmarks like Llama and Qwen on compute, storage, and network hardware.

• Collaborate with frameworks including Megatron-LM, TorchTitan, Composer, vLLM, and llama.cpp.

• Design and optimize distributed serving architectures, including sharded/replicated, tensor/pipeline parallelism, and MoE expert placement.

• Enhance continuous/adaptive batching, KV-cache sharding/offloading, and prefix caching strategies.

• Validate RDMA/GPUDirect RDMA, congestion control, and tradeoffs for collective/point-to-point communication during inference.

• Create proof points that compare Cornelis Omni-Path with competing interconnects.

• Convert performance metrics into compelling narratives for sales, marketing, and customer engagement.

• Instrument and visualize performance metrics using tools such as Nsight Systems, VTune, perf, application timers, Prometheus, and Grafana.

• Promote best practices through briefs, READMEs, blogs, articles, and peer-reviewed publications.

• Collaborate with architecture, firmware, software teams, and lighthouse customers to translate lab results into successful field implementations.

⛳️ Requirements

• Bachelor's degree in Computer Science, Electrical Engineering, Computer Engineering, Mathematics, or a related field (Master's preferred).

• 2–3+ years of experience running AI/ML workloads at the cluster scale.

• Demonstrated ability to set up, execute, and analyze AI benchmarks.

• Strong understanding of message passing, collective operations, scaling efficiency, and identifying bottlenecks for training and inference.

• Hands-on experience with distributed training beyond single GPU (DP/TP/PP, ZeRO, FSDP, sharded optimizers).

• Familiarity with distributed inference architectures (replicated vs sharded, tensor/KV parallel, MoE).

• Practical experience with PyTorch, DeepSpeed, Megatron-LM, RCCL/NCCL, vLLM, and llama.cpp.

• Proficient with GCC/LLVM/Intel/OneAPI compilers and MPI stacks.

• Proficient in Python and shell scripting.

• Knowledge of Omni-Path/OPA, InfiniBand, Ethernet/RDMA/RoCE, and Linux performance tuning.

• Understanding of NIC offloads, CQ moderation, pacing, and ECN.

• Excellent written and verbal communication skills.

• Preferred: Experience with cluster schedulers (SLURM, PBS).

• Preferred: GPU/communications profiling and tracing experience using Nsight Systems, Nsight Compute, ROCm tools/rocprof/roctracer, VTune, or perf.

• Preferred: Background in HPC performance engineering or storage solutions (BeeGFS, Lustre, NVMe-oF).

🏝️ Benefits

• Full-Time position (U.S. Benefit Eligible).

• Fully supported remote work for employees based in the United States.

• Occasional travel to partner and client meetings, conferences, and company in-person collaborations (less than 10%).