Head of Infrastructure Operations

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee and manage global infrastructure operations across pod sites in the US and Europe, encompassing site bring-up, rack and stack, cabling, power and cooling integration, burn-in, acceptance, and transitioning to steady-state operations and decommissioning.

• Take full ownership of the server, storage, and network layers from end to end, which includes Linux fleet management, bare-metal provisioning and imaging, firmware and driver lifecycle management, high-throughput storage, and data center as well as inter-site networks.

• Define and refine repeatable, documented, and automated site-launch playbooks for new 1 MW deployments.

• Manage capacity planning, monitoring, alerting, on-call support, incident response, and post-incident evaluations.

• Focus on reducing GPU, node, and network failure rates.

• Recruit, lead, and develop data center, systems, and network engineers across various time zones.

• Handle colocation, power, connectivity, and hardware vendors and partners, including managing capacity contracts, lead times, spares, RMA flows, and site-level budgets.

• Collaborate with hardware and platform engineering teams to ensure designs are scalable and integrate field learnings into future servers and pods.

• Utilize LLM-based agents to automate runbooks, triage processes, and tooling.

• Ensure operational security and compliance for physical infrastructure, which includes access control, hardening, asset tracking, audit readiness, and data residency requirements.

• Report directly to the Head of AI Infrastructure and work together with hardware, platform, and inference engineering teams as well as external partners.

⛳️ Requirements

• Over 15 years of experience in infrastructure engineering and operations, with several years leading the function at a company that operates its own hardware at scale.

• Proven history of constructing infrastructure operations across multiple sites, regions, or countries.

• Experience in establishing data centers, pods, or colocation facilities.

• In-depth, up-to-date, hands-on expertise in Linux, covering kernel and driver behavior, performance tuning, bare-metal provisioning, configuration management, and fleet automation.

• Strong fundamentals in storage, including NVMe, distributed and parallel file systems, object storage, and high-throughput and IOPS design.

• Solid understanding of networking principles, such as L2/L3 design, BGP, data center fabrics, high-bandwidth interconnects, and network troubleshooting.

• Demonstrated ability to lead high-performance teams, recruit, mentor, and retain engineers, establish standards, and foster on-call and incident response cultures.

• Regular use of AI for scripting, runbooks, analysis, documentation, and troubleshooting.

• Capability to take ownership of outcomes in a dynamic and ambiguous environment and make decisions with incomplete information.

• Willingness to travel to sites in the US and Europe as required.

• Nice to have: experience managing large GPU fleets for AI training or inference, including familiarity with NVIDIA drivers, Fabric Manager, and NCCL stacks.

• Nice to have: practical experience with RDMA fabrics, InfiniBand or RoCE, 400G+ Ethernet, and monitoring tools.

• Nice to have: experience with liquid-cooled, high-density, modular, or containerized data centers and site-level power and cooling integration.

• Nice to have: knowledge of hardware supply chain dynamics, including server ODMs, component lead times, inventory management, spares, and RMA economics.

• Nice to have: familiarity with automation stacks such as Ansible, Terraform, PXE, image pipelines, IPMI, Redfish, Prometheus, and Grafana or similar tools.

• Nice to have: experience with Kubernetes or Slurm on bare metal.

• Nice to have: infrastructure experience that meets data residency, sovereignty, or compliance requirements.

🏝️ Benefits

• Meaningful stock options.

• Remote-first work environment.

• Biannual in-person meetings.

• Flexible scheduling outside of core collaboration hours.

• Adequate downtime following rapid release cycles.

• Generous paid time off, including vacation, sick days, and public holidays.

• Paid maternity, paternity, and caregiver leave.

• Company retreats held twice a year in inspiring locations.