Senior Engineer

This is a fully remote position, open to applicants in California, +2 more states.

📋 Description

• Design and develop systems that transform cluster validation into a repeatable and verifiable software process.

• Create and implement cluster readiness and certification tools using Go and Python on Kubernetes and Slurm, including controllers, operators, command-line interfaces, and scheduler integrations.

• Establish a catalog of benchmarks and actual workloads covering communication, computing, storage, diagnostics, training, inference, reliability, and security.

• Develop provenance and evidence pipelines that capture execution details and generate signed, auditable results.

• Integrate new GPU platforms, schedulers, and fabrics into testing, from single-node setups to multi-node production cluster validation.

• Collaborate with internal platform, networking, and solutions teams, as well as cloud partners, to identify and resolve performance and correctness issues.

• Contribute to the open-source Cluster Readiness Engine and associated projects by creating design documents, documentation, and conducting reviews.

• Write design documents, runbooks, and operational guides for cluster validation and result analysis.

⛳️ Requirements

• A BS, MS, or Ph.D. in Computer Science, Computer/Electrical Engineering, or a related technical discipline, or equivalent experience.

• Over 8 years of software engineering experience focused on building infrastructure or systems software, preferably for large-scale computing, cloud, or HPC environments.

• Proficient programming skills in Go and Python, with a track record of delivering production-quality services, controllers, or command-line tools.

• Extensive knowledge of Linux systems, containers, and Kubernetes, including experience writing operators or controllers using the Kubernetes API.

• Practical understanding of HPC schedulers like Slurm and fundamentals of GPU clusters: drivers, NCCL, InfiniBand or RoCE fabrics, and GPU health monitoring tools such as DCGM.

• Experience in benchmarking or validating distributed systems, with a strong focus on measurement, reproducibility, and evidence.

• Proven ability to conduct complex technical investigations to identify root causes and effectively communicate findings to engineering and leadership teams.

• Hands-on experience with large-scale distributed training or inference workloads using frameworks like PyTorch, Megatron, NeMo, vLLM, SGLang, or TensorRT-LLM.

• Familiarity with the NVIDIA ecosystem, including DGX systems, GB200/GB300 platforms, CUDA, NVLink and NVSwitch fabrics, and NVIDIA networking.

• Previous contributions to open-source infrastructure projects, particularly Kubernetes operators, scheduler plugins, or benchmarking frameworks.

• Background in software supply-chain security, attestation, or signed artifacts, along with the ability to incorporate audit-grade evidence into systems.

• Knowledge of observability stacks (Prometheus, Grafana, OpenTelemetry), GitOps workflows, and infrastructure as code practices for GPU clusters.

🏝️ Benefits

• Equity

• Benefits

• Equal opportunity employment