
Senior Technical Support Engineer – Slurm
Posted 1 day ago

Posted 1 day ago
This is a fully remote position, open to applicants in United States.
• Take charge of Slurm support cases from the initial investigation to resolution for clients utilizing production AI and HPC clusters.
• Diagnose intricate issues involving slurmctld, slurmd, slurmdbd, job scheduling, node management, resource allocation, accounting, authentication, and high availability.
• Resolve Slurm configuration and policy features, which include partitions, reservations, priorities, fair-share, quality of service, backfill, preemption, GRES/TRES, cgroups, and job constraints.
• Examine performance, reliability, and scalability concerns by utilizing logs, diagnostic data, configuration analysis, reproductions, and source-level debugging when necessary.
• Identify issues across Slurm and its dependencies, such as Linux, MUNGE, databases, networking, parallel storage, containers, GPUs, and cluster-management systems.
• Provide guidance to customers on Slurm configuration, upgrades, operational practices, system resource management, and safe recovery from production incidents.
• Work in collaboration with engineering teams by creating clear technical descriptions, reproducible test cases, and well-documented defect reports.
• Produce guides, knowledge-base articles, diagnostic tools, and internal training to enhance Slurm expertise within the support organization.
• Bachelor's degree in Computer Science, Engineering, or a related discipline, or equivalent experience.
• Over 5 years of practical experience in administering and supporting Slurm in production HPC or AI settings, including handling business-critical outage incidents.
• Expert-level comprehension of Slurm architecture, daemons, configuration, scheduling behavior, accounting, resource management, and failure modes.
• Ability to independently identify complex Slurm incidents and guide them to a technically sound resolution.
• Extensive Linux system administration and troubleshooting experience, including systemd, cgroups, authentication, networking, and database-backed services.
• Proficiency in operating Slurm across multi-user clusters with intricate scheduling policies and diverse compute resources.
• Strong analytical and research capabilities, demonstrating proficiency in differentiating Slurm defects from configuration, integration, infrastructure, and workload issues.
• Exceptional written and verbal communication skills, including the ability to translate detailed technical findings into clear explanations and actionable recommendations.
• Experience in supporting large-scale Slurm environments comprising thousands of nodes or GPUs.
• Proven experience in diagnosing scheduler performance, job throughput, controller load, and database scaling issues.
• Familiarity with Slurm source code, plugins, SPANK, Lua job-submit plugins, or upstream issue investigation.
• Experience with containers and HPC integration technologies such as Pyxis, Enroot, Apptainer, or Singularity.
• Previous experience integrating Slurm with NVIDIA Base Command Manager, Bright Cluster Manager, or another cluster-management platform.
• Highly competitive salaries.
• Comprehensive benefits package.
• Equity.
• Benefits for you and your family.
Spherica
Akamai Technologies
Cohere Health
Get handpicked remote jobs straight to your inbox weekly.