Member of Technical Staff, GPU Infrastructure

This is a fully remote position, open to applicants in United States, +4 more locations.

📋 Description

• Deploy and manage GPU Kubernetes clusters across different cloud environments.

• Bring up new generations of hardware and diverse providers.

• Collaborate with cloud service providers and hardware partners to enhance reliability, performance, and cost efficiency.

• Address challenges related to RDMA and high-speed interconnects.

• Automate the detection and remediation of node, GPU, and fabric faults.

• Optimize fleet utilization and monitor associated costs.

• Reduce the turnaround time for faulty nodes to be restored to service.

• Diagnose production failures and performance issues.

• Enhance the operability of systems.

• Work together with AI researchers and ML engineers to ensure stability, scalability, and observability.

• Take part in a compensated 24x7 on-call rotation.

⛳️ Requirements

• Proven experience in operating GPU Kubernetes clusters with thousands of GPUs for ML or HPC workloads.

• Proficient in debugging issues related to Linux, networking, and performance.

• Familiarity with RDMA, NCCL, or understanding accelerator health is advantageous.

• Background in software development using Python or Go.

• Experience in extending established open-source projects.

• Ability to work collaboratively with researchers, ML engineers, or teams that have varying objectives.

• Skilled in identifying bottlenecks, proposing solutions, and driving implementations in a dynamic environment.

• Availability to participate in a compensated 24x7 on-call rotation.

• Must be authorized to work in the current country of residence.

• Must disclose whether work sponsorship is needed now or in the future.

🏝️ Benefits

• A weekly lunch stipend of $75/£75 or the equivalent in local currency.

• Comprehensive health and dental benefits.

• Dedicated mental health budget.

• RRSP matching.

• 401K plan.

• Pension Scheme.

• 100% parental leave top-up for up to 6 months, applicable to either parent.

• Annual enrichment benefits for arts and culture, fitness/wellness, quality time, and workspace enhancement.

• Education and learning stipend for conferences, courses, and coaching opportunities.

• 6 weeks of paid vacation (30 working days).

• Budget allocated for travel to other offices if working remotely.

• Annual company offsite event.

• Co-working benefit available for employees not located near an office.

• $500 stipend for home office setup.

• Daily lunch program, snacks, and regular community and social events for office-based employees.