Technical Lead – GPU Infrastructure

This is a fully remote position, open to applicants in Egypt.

📋 Description

• Take full responsibility for the architecture of the Cosmic AC GPU compute and managed inference platform from start to finish.

• Create and sustain architecture proposals, high-level designs, and low-level designs through thorough reviews.

• Oversee and manage a team of around twelve distributed engineers across various domains, including backend, frontend, DevOps, QA, and documentation.

• Set engineering standards, perform code and design reviews, manage release gates, conduct one-on-one meetings, and provide feedback for growth and performance.

• Design, develop, and manage a Slurm service tailored for research users.

• Handle the Slurm controller, accounting, partitions, login nodes, node onboarding, driver and CUDA baselines, health monitoring, autohealing, storage visibility, identity, and isolation.

• Manage the bootstrap and lifecycle of Kubernetes clusters on partner-supplied bare metal.

• Operate NVIDIA GPU Operator and Network Operator, KubeVirt, and implement VFIO-based GPU isolation, upgrades, backup, recovery, and node replacement.

• Architect managed inference at scale, covering serving architecture, multi-GPU and multi-node parallelism, autoscaling, request routing, endpoint reliability, and confidential compute-capable capacity.

• Define metrics, logging, alerting, and SLOs across the control plane, GPU fleet, and application tiers.

• Lead incident response efforts, conduct post-incident reviews, and implement a sustainable on-call model.

• Serve as the primary technical liaison to infrastructure partners and vendors.

• Convert requirements into written specifications and acceptance tests, handle escalations, and assist in capacity planning and hardware sourcing.

• Collaborate with research, model-training, and product teams to translate workloads into platform requirements and negotiate capacity.

• Complete the platform team and establish the technical standards for new engineers.

⛳️ Requirements

• A minimum of eight years of hands-on engineering experience.

• At least three years of experience leading teams responsible for building and maintaining infrastructure platforms relied upon by other teams.

• A Bachelor's or Master's degree in computer science or engineering, or equivalent practical experience.

• Direct experience operating Slurm at scale, including components such as slurmctld, slurmdbd, partitions, QoS, priority, accounting, prolog and epilog, health scripting, and upgrades while jobs are running.

• Ideal candidates will have experience managing an HPC or GPU training cluster for a research community.

• Experience with operating NVIDIA GPU fleets on bare metal, including managing the driver and CUDA lifecycle, Fabric Manager, NVSwitch, DCGM, MIG, node burn-in, and acceptance testing.

• Familiarity with InfiniBand, subnet configuration, RDMA, SR-IOV, and troubleshooting multi-node NCCL performance issues.

• Extensive knowledge of Linux systems, including kernel modules, drivers, PCIe passthrough, vfio-pci, cgroups, namespaces, and performance optimization.

• Proficient in production Kubernetes operations, including control plane management, upgrades, CNI, CSI, operators, custom controllers, and designing for multi-tenancy.

• Experience with HPC storage and data transfer, including technologies such as VAST, Lustre, NFS, node-local NVMe caching, and the distribution of large model weights and datasets.

• Familiarity with monitoring tools such as Prometheus, Grafana, Loki or similar, as well as SLOs, incident response, and post-incident analysis.

• Working knowledge of JavaScript and Node.js sufficient for reviewing control-plane, CLI, and worker services and making architecture decisions.

• Experience deploying a platform with real users, such as multi-tenant IaaS/PaaS or research computing services.

• Competence in managing teams across different time zones, conducting cross-track reviews, documenting architecture decisions, and effectively challenging partners or executives with rationale.

• Excellent written and spoken English communication skills.

• Must be available to work within the UTC to UTC+5:30 time zones.

• Desirable: Experience with Slurm operators on Kubernetes or Kubernetes-native schedulers.

• Desirable: Familiarity with modern serving stacks such as vLLM, SGLang, and TensorRT-LLM.

• Desirable: Knowledge of VM/container isolation, confidential computing, Cluster API, kubeadm, Cilium, autohealing, infrastructure as code, GitOps, GPU cloud/HPC/AI lab experience, distributed systems, and partnerships with hardware providers.

🏝️ Benefits

• Fully remote work arrangement.

• Occasional travel to partner locations and team events.