Staff AI Scheduling, Orchestration Engineer

This is a fully remote position, open to applicants in California, +1 more state.

📋 Description

• Create and execute sophisticated batch scheduling frameworks utilizing tools like Volcano or YuniKorn to facilitate multi-node gang scheduling.

• Design and oversee cluster-wide admission control along with advanced job queueing systems using Kueue.

• Utilize Kubernetes Dynamic Resource Allocation (DRA) and custom scheduler plugins to handle intricate accelerator requests.

• Develop topology-aware pod placement strategies that are optimized for low-latency communication through NVLink and InfiniBand technologies.

• Implement automated GPU sharing solutions, including MIG and time-slicing, along with policies for multi-tenancy isolation.

• Work closely with GPU Systems and Storage teams to align scheduling with bare-metal hardware and storage I/O patterns.

• Enhance the reliability and scalability of the scheduling infrastructure, addressing resource contention and deadlock issues in large-scale HPC settings.

• Mentor junior engineers and perform design reviews to uphold architectural standards in the orchestration layer.

⛳️ Requirements

• Bachelor’s or Master’s degree in Computer Science, Electrical Engineering, or a related discipline.

• Over 6 years of experience in distributed systems engineering.

• In-depth, hands-on knowledge of Kubernetes scheduling frameworks and orchestrators.

• Significant experience with AI workload execution patterns and distributed training frameworks, such as PyTorch Distributed, Ray, and MPI.

• Proven ability to operate, debug, and scale scheduling systems in high-performance computing (HPC) or large-scale production cloud settings.

• Strong understanding of GPU hardware architectures and the scheduling challenges associated with distributed AI training and inference.

• Experience in infrastructure automation and infrastructure-as-code, including tools like Terraform and Go-based Operators.

• Exceptional technical communication and leadership abilities.

• Capability to influence cross-functional teams and align architectural objectives.

• Ability to thrive in a fast-paced engineering environment and translate complex, ambiguous requirements into clear, scalable engineering solutions.

🏝️ Benefits

• Competitive salary and performance-based incentives.

• Comprehensive health, dental, and vision insurance.

• Flexible working hours and remote work options.

• Opportunities for professional development and continuous learning.

• A collaborative and innovative work environment.