Technical Support Engineer – Slurm

This is a fully remote position, open to applicants in India.

📋 Description

• Take ownership of Slurm support cases from the initial investigation to resolution for clients operating production AI and HPC clusters.

• Troubleshoot intricate issues related to slurmctld, slurmd, slurmdbd, job scheduling, node management, resource allocation, accounting, authentication, and ensuring high availability.

• Address Slurm configuration and policy challenges involving partitions, reservations, priorities, fair-share, quality of service, backfill, preemption, GRES/TRES, cgroups, and job constraints.

• Analyze performance, reliability, and scalability concerns utilizing logs, diagnostic data, configuration assessments, reproduction efforts, and source-level debugging as necessary.

• Identify issues across Slurm and its dependencies, including Linux, MUNGE, databases, networking, parallel storage, containers, GPUs, and cluster management systems.

• Provide guidance to customers on Slurm configuration, upgrades, operational practices, resource management, and safe recovery procedures from production incidents.

• Work collaboratively with engineering teams by providing clear technical descriptions, reproducible test cases, and well-documented defect reports.

• Create guides, knowledge base articles, diagnostic tools, and internal training materials to enhance Slurm expertise throughout the support organization.

⛳️ Requirements

• Bachelor’s degree in Computer Science, Engineering, or a related discipline, or equivalent practical experience.

• More than 5 years of practical experience administering and supporting Slurm in production HPC or AI settings, including handling business-critical outage incidents.

• Advanced understanding of Slurm architecture, daemons, configuration, scheduling behavior, accounting, resource management, and failure modes.

• Capability to independently diagnose complex Slurm incidents and lead them to sound technical resolutions.

• Extensive experience in Linux system administration and troubleshooting, including systemd, cgroups, authentication, networking, and database-backed services.

• Experience managing Slurm in multi-user clusters with intricate scheduling policies and diverse computing resources.

• Strong analytical and research skills, especially in differentiating Slurm defects from configuration, integration, infrastructure, and workload challenges.

• Exceptional written and verbal communication abilities, particularly in transforming detailed technical findings into clear explanations and actionable recommendations.

• Experience in supporting large-scale Slurm environments that consist of thousands of nodes or GPUs.

• Proficiency in diagnosing scheduler performance, job throughput, controller load, and database scaling issues.

• Familiarity with Slurm source code, plugins, SPANK, Lua job-submit plugins, or upstream issue investigation.

• Experience with containers and HPC integration technologies such as Pyxis, Enroot, Apptainer, or Singularity.

• Prior experience integrating Slurm with NVIDIA Base Command Manager, Bright Cluster Manager, or any other cluster management platform.

🏝️ Benefits

• A diverse and supportive work environment.

• Opportunities to create a lasting impact on the world.

• Full-time employment.