Forward Deployed Infrastructure Engineer

This is a fully remote position, open to applicants in United States.

📋 Description

• Develop, validate, and benchmark new customer clusters, then provide them with comprehensive documentation.

• Deliver in-depth escalations concerning multinode and NCCL performance, GPU and hardware malfunctions, driver and fabric issues, containers, and scheduling systems.

• Identify fault locations, escalate problems to infrastructure providers with supporting evidence, and coordinate maintenance schedules, RMAs, solutions for hung nodes, and disputed fault attribution.

• Manage assigned customer accounts and gain an understanding of their workloads and infrastructure.

• Advise clients on necessary changes to prevent potential issues.

• Take ownership of monitoring and alerting for customer clusters utilizing Grafana and Prometheus.

• Automate repetitive tasks and transform escalations into L1 runbooks.

• Remain embedded within customer environments following cluster handoff.

⛳️ Requirements

• Extensive Linux experience and complete comfort in the command line interface, even in challenging environments.

• Practical experience with multinode GPU setups: NCCL, InfiniBand or RoCE, collective performance debugging, topology, and placement.

• Capability in hardware fault triage on GPU nodes, including handling XID and ECC errors, dmesg, lspci, nvidia-smi, thermal, and power issues.

• Proficient in provisioning and operating GPU clusters using Kubernetes, Slurm, or both.

• Genuine ease in customer-facing interactions, including the delivery of difficult news and confidently assigning responsibility.

• Sound judgment in determining when to continue investigation and when to escalate issues.

• Experience with parallel filesystems (Weka, Lustre, GPFS) and high-performance storage is preferred.

• Familiarity with Grafana, Prometheus, or similar observability tools in production settings is preferred.

• Previous experience as a forward deployed engineer, solutions architect, or technical account manager is preferred.

• Knowledge of infrastructure as code (Terraform, Ansible) and CI practices for cluster provisioning is preferred.

• Exposure to model training or inference workloads from the practitioner's perspective is preferred.

• Legally authorized to work in the United States.

• Must indicate whether employment visa sponsorship is needed.

🏝️ Benefits

• Equal opportunity employer.

• Inclusive work environment.

• Remote work arrangement.