Tech Lead Manager, GPU Cluster Infrastructure

This is a fully remote position, open to applicants in California.

📋 Description

• Establish the technical vision for the platform and take responsibility for its roadmap.

• Determine which systems to operate, manage scheduling and storage across providers, and identify key metrics for measurement.

• Lead the design of architecture, scheduling, and storage systems.

• Troubleshoot failures across various layers, including node health, GPU and fabric issues, and multi-node job stalls.

• Recruit and develop a small team of senior engineers.

• Prioritize tasks, define ownership, scope projects, and offer regular feedback and coaching.

• Set the security framework for a shared cluster where AI agents conduct experiments, focusing on identity and access, workload isolation, and sandboxing.

• Outline operational procedures, including on-call rotations, incident response, postmortems, fault tolerance, and observability.

• Engage in operational practices alongside the team.

• Act as the escalation point for research teams and infrastructure providers.

• Transform recurring issues into effective platform solutions.

• Collaborate directly with researchers and engineers to ensure large-scale experiments are efficient and resilient.

⛳️ Requirements

• Proven experience in leading engineers as a manager, tech lead, or project lead, encompassing the setting of technical direction, project scoping, and feedback provision.

• A strong desire to manage people directly; previous experience with direct reports is not a prerequisite.

• Over 5 years of experience in systems or infrastructure engineering within a production Linux environment.

• Experience in managing GPU, HPC, or large-scale batch processing platforms.

• Proven track record of owning systems from the design phase through to operational execution.

• In-depth knowledge in at least one area such as scheduling, storage, networking, security, or GPU systems, along with sufficient breadth to evaluate designs in other areas.

• Experience managing production Kubernetes for GPU workloads with a batch processing layer like Slurm, Kueue, or Volcano, including management of quotas, priority and preemption, and node health.

• Familiarity with infrastructure as code and observability practices for a production fleet.

• Proficiency with tools such as Terraform or Ansible, Helm, ArgoCD, and Prometheus, or their equivalents.

• Strong programming skills in languages like Python, Go, Rust, C++, or another commonly utilized infrastructure language.

• Ability to produce clear technical documentation for engineers, researchers, and service providers.

• Additional desirable skills may include expertise in distributed training infrastructure, distributed storage, cluster security, scheduler internals, multi-provider platforms, and building teams from the ground up.

🏝️ Benefits

• Health insurance with 94% of the premium covered by the organization, effective within one month of the start date.

• 401(k) plan featuring up to a 2% match.

• Accrual of 25 days of paid time off per year, calculated weekly.

• Up to 10 days of paid sick leave annually.

• Paid leave for bereavement, family, medical, and pregnancy disability reasons.

• Provision of a work computer and WFH stipend for eligible employees.

• Catered lunches and dinners provided on workdays at the Berkeley office.

• Visa sponsorship available for employees working in-person.

• Paid work trial lasting up to one week.