Technical Lead – GPU Infrastructure

This is a fully remote position, open to applicants in Egypt.

📋 Description

• Take ownership of the complete architecture for Tether Data's Cosmic AC GPU compute and managed inference platform.

• Lead the development of architecture proposals, high-level and low-level designs, conduct reviews, and maintain baselines.

• Manage and oversee approximately twelve distributed engineers across backend, frontend, DevOps, QA, and documentation teams.

• Design, build, and maintain a managed Slurm service for research users.

• Oversee cluster bootstrap and lifecycle on partner-provided bare metal, including NVIDIA GPU and Network Operators, VM-based GPU isolation, and day-2 operations.

• Own the architecture for managed inference serving, including multi-GPU and multi-node parallelism, autoscaling, request routing, endpoint reliability, and confidential-compute-capable capacity.

• Establish metrics, logging, alerting, SLOs, incident response protocols, post-incident reviews, and a sustainable on-call model.

• Act as the primary technical liaison to infrastructure partners and vendors; define specifications, acceptance tests, escalations, capacity planning, and input for hardware sourcing.

• Convert research, model-training, and product workloads into platform requirements and manage capacity.

• Recruit and finalize the platform team while setting high technical standards.

• Ensure the platform is delivered within a specified six-month timeframe.

⛳️ Requirements

• A minimum of eight years of hands-on engineering experience.

• At least three years of experience leading teams responsible for building and maintaining infrastructure platforms relied upon by other teams.

• A Bachelor's or Master's degree in computer science or engineering, or equivalent practical experience.

• Practical experience running Slurm at scale, including slurmctld and slurmdbd, partitions, QoS and priority settings, accounting, prolog and epilog, node health scripting, and upgrades with active jobs.

• Ideally, experience operating an HPC or GPU training cluster for a research audience.

• Experience managing a bare-metal NVIDIA GPU fleet, including driver and CUDA lifecycle, Fabric Manager, NVSwitch, DCGM, MIG, and node burn-in and acceptance processes.

• Knowledge of InfiniBand fabric and subnet configuration, RDMA, SR-IOV, and troubleshooting multi-node NCCL performance issues.

• Expertise in Linux systems, covering kernel modules and drivers, PCIe passthrough, vfio-pci, cgroups, namespaces, and performance optimization.

• Experience with production Kubernetes operations, including control plane management, upgrades, CNI, CSI, operators, custom controllers, and multi-tenancy design.

• Familiarity with HPC storage and data movement utilizing shared filesystems such as VAST, Lustre, and NFS, node-local NVMe caching, and the distribution of large model weights and datasets.

• Proficiency in observability and operations tools like Prometheus, Grafana, Loki or similar, along with SLOs, incident response, and post-incident review processes.

• Working proficiency in JavaScript and Node.js for evaluating and making architectural decisions regarding control plane, CLI, and worker services.

• Proven experience delivering a platform used by real users, such as multi-tenant IaaS/PaaS or research computing services.

• Experience in people management across different time zones, conducting cross-track reviews, documenting architecture decisions, and the ability to communicate technical decisions to partners and executives.

• Excellent proficiency in written and spoken English.

• Desirable experience includes knowledge of Slurm operators, Kubernetes-native schedulers, modern serving stacks, VM/container isolation, confidential computing, Cluster API, kubeadm, Cilium, infrastructure as code, GitOps, GPU cloud/HPC/AI lab platforms, distributed systems, and partnerships with hardware providers.

🏝️ Benefits

• Remote work opportunities.

• Occasional travel to partner sites and team events.