Technical Lead – GPU Infrastructure

This is a fully remote position, open to applicants in Ireland.

📋 Description

• Take ownership of the comprehensive architecture for Tether Data's Cosmic AC GPU compute and managed inference platform.

• Develop and uphold architecture proposals, high-level designs, and low-level designs through rigorous review processes.

• Lead and manage a team of approximately twelve distributed engineers across backend, frontend, DevOps, QA, and documentation.

• Establish engineering standards, perform code and design reviews, oversee release gates, conduct one-on-ones, and provide feedback on growth and performance.

• Design, construct, and manage a Slurm service tailored for research users.

• Oversee controller and accounting, partitions and login nodes, node onboarding, driver and CUDA baselines, detection of stalled jobs and node health, draining, autohealing, storage visibility, identity, and isolation.

• Manage the bootstrap and lifecycle of the Kubernetes cluster on partner-supplied bare metal.

• Supervise the NVIDIA GPU Operator, Network Operator, VM-based GPU isolation, KubeVirt, VFIO, upgrades, backup and recovery, and node replacement.

• Lead the managed inference architecture, ensuring multi-GPU and multi-node parallelism, autoscaling, request routing, endpoint reliability, and confidential-compute-capable capacity.

• Implement metrics, logging, alerting, and SLOs across control plane, GPU fleet, and application tiers.

• Direct incident response efforts, conduct post-incident reviews, and develop a sustainable on-call model.

• Act as the primary technical liaison to infrastructure partners and vendors.

• Convert requirements into detailed written specifications and acceptance tests, manage escalations, and assist in capacity planning and hardware sourcing.

• Collaborate with research, model-training, and product teams to translate workloads into platform requirements and negotiate limited capacity.

• Complete the platform team and set a high technical standard for new engineers.

• Take charge of implementation and delivery plans within a defined initial six-month delivery timeframe.

⛳️ Requirements

• A minimum of eight years of hands-on engineering experience.

• At least three years of experience in leading teams that build and maintain infrastructure platforms relied upon by other teams.

• A Bachelor's or Master's degree in computer science or engineering, or equivalent practical experience.

• Demonstrated hands-on experience operating Slurm at scale, including slurmctld, slurmdbd, partitions, QoS, priority, accounting, prolog and epilog, node health scripting, and performing upgrades while jobs are running.

• Ideally, experience managing an HPC or GPU training cluster for a research community.

• Proven experience operating NVIDIA GPU fleets on bare metal, encompassing driver and CUDA lifecycle, Fabric Manager, NVSwitch, DCGM, MIG, node burn-in, and acceptance processes.

• Familiarity with InfiniBand, subnet configuration, RDMA, SR-IOV, and troubleshooting multi-node NCCL performance issues.

• In-depth knowledge of Linux systems, including kernel modules and drivers, PCIe passthrough, vfio-pci, cgroups, namespaces, and performance optimization.

• Experience with production Kubernetes operations, including control plane, upgrades, CNI, CSI, operators, custom controllers, and multi-tenancy design.

• Knowledge of HPC storage and data movement technologies, including VAST, Lustre, NFS, node-local NVMe caching, and distributing large model weights and datasets.

• Familiarity with Prometheus, Grafana, Loki or similar tools, as well as SLOs, incident response, and post-incident reviews.

• Proficiency in JavaScript and Node.js sufficient for reviewing control plane, CLI, and worker services and making architectural decisions.

• Experience delivering a multi-tenant IaaS, PaaS, or research computing service with resource isolation, quotas, usage metering, and user-friendly API and CLI interfaces.

• Proven ability in people management across time zones, cross-track reviews, written architectural decisions, and the capability to challenge partners or executives with well-founded reasoning.

• Excellent command of written and spoken English.

• Fully remote work location allowed, with a requirement to be situated between UTC and UTC+5:30.

• Willingness to undertake occasional travel to partner sites and team events.

• Desirable experience with Slurm operators on Kubernetes or Kubernetes-native schedulers.

• Desirable experience with modern serving stacks such as vLLM, SGLang, and TensorRT-LLM.

• Desirable experience with VM and container isolation, confidential computing, Cluster API, kubeadm, Cilium, NVSentinel-class autohealing, infrastructure as code, GitOps, GPU cloud/HPC/AI lab platforms, distributed systems, and relationships with hardware providers.

🏝️ Benefits

• Fully remote work environment.

• Opportunities for occasional travel to partner sites and team events.