Forward Deployed Engineer – SRE

This is a fully remote position, open to applicants in California.

📋 Description

• Act as the main technical liaison for teams managing extensive training and inference tasks.

• Collaborate within customer settings to troubleshoot actual failures: NCCL timeouts, stragglers, checkpoint I/O stalls, degraded links, and OOM patterns.

• Analyze and enhance distributed training performance on active workloads.

• Take ownership of reliability outcomes for the accounts assigned to you.

• Monitor the health and performance of high-speed interconnects (InfiniBand, RoCE, NVLink).

• Lead incident response efforts for intricate, multi-layer failures involving hardware, networking, orchestration, and ML frameworks.

• Transform every recurring deployment issue into automated solutions.

⛳️ Requirements

• Practical experience in operating GPU clusters in production environments (NVIDIA A100/H100/H200/B200 or equivalent).

• Hands-on experience with InfiniBand, RoCE, or NVLink fabrics related to distributed training.

• Understanding of how large training and inference jobs function in practice.

• Advanced Linux skills: including kernel tuning, driver management (NVIDIA drivers, CUDA toolkit), cgroup/namespace internals, container runtimes, and performance profiling at both the syscall and hardware levels.

• Significant experience running Kubernetes in production with GPU workloads.

• Strong engineering expertise in Python, Go, or Bash.

• Proficiency in Infrastructure-as-Code (Terraform, Helm, Ansible, or equivalent).

• Experience in developing monitoring and alerting systems for GPU-specific telemetry.

🏝️ Benefits

• Comprehensive benefits package: covering you and your dependents, including healthcare, dental, and vision insurance.

• 401(k) plan.

• Unlimited paid time off (PTO).