Technical Lead – GPU Infrastructure

This is a fully remote position, open to applicants in Romania.

📋 Description

• Take complete ownership of the Cosmic AC platform architecture from start to finish, which includes crafting architecture proposals and developing both high-level and low-level designs.

• Lead and manage a team of approximately twelve engineers distributed across backend, frontend, DevOps, QA, and documentation disciplines.

• Establish engineering standards, perform code and design reviews, manage release gates, facilitate one-on-one meetings, and provide input on growth and performance.

• Design, create, and maintain a managed Slurm service tailored for research users.

• Oversee controller and accounting functions, partitions and login nodes, node onboarding, driver and CUDA baselines, stalled-job and node-health detection, drainage and autohealing, storage visibility, as well as identity and isolation.

• Manage the bootstrap and lifecycle of Kubernetes clusters on partner-supplied bare metal infrastructure.

• Implement and manage NVIDIA GPU Operator, Network Operator, VM-based GPU isolation with KubeVirt and VFIO, along with upgrades, backup, recovery, and node replacement processes.

• Own the architecture for managed inference, ensuring multi-GPU and multi-node parallelism, autoscaling, request routing, endpoint reliability, and capacity for confidential computing.

• Establish metrics, logging, alerting, and Service Level Objectives (SLOs) across the control plane, GPU fleet, and application tiers.

• Lead incident response initiatives, conduct post-incident reviews, and develop a sustainable on-call model.

• Act as the primary technical liaison to infrastructure partners and vendors.

• Transform requirements into detailed written specifications and acceptance tests, manage escalations to resolution, and contribute to capacity planning and hardware procurement.

• Translate research, model-training, and product workloads into platform requirements, while negotiating capacity when necessary.

• Complete the platform team and set a high technical standard for new engineers.

⛳️ Requirements

• A minimum of eight years of hands-on engineering experience.

• At least three years of experience in leading teams that build and manage infrastructure platforms.

• A Bachelor's or Master's degree in computer science or engineering, or equivalent practical experience.

• Practical experience operating Slurm at scale, including slurmctld, slurmdbd, partitions, QoS, priority, accounting, prolog and epilog, node health scripting, and managing upgrades while jobs are running.

• Ideal candidates will have experience running HPC or GPU training clusters for research users.

• Familiarity with operating NVIDIA GPU fleets on bare metal, including managing driver and CUDA lifecycles, Fabric Manager, NVSwitch, DCGM, MIG, node burn-in, and acceptance processes.

• Knowledge of InfiniBand, subnet configuration, RDMA, SR-IOV, and troubleshooting multi-node NCCL performance challenges.

• Extensive knowledge of Linux systems, including kernel modules, drivers, PCIe passthrough, vfio-pci, cgroups, namespaces, and performance optimization techniques.

• Experience with production Kubernetes operations, covering control plane, upgrades, CNI, CSI, operators, custom controllers, and multi-tenancy aspects.

• Familiarity with HPC storage and data movement, including VAST, Lustre, NFS, node-local NVMe caching, and distributing large model weights and datasets.

• Experience with tools like Prometheus, Grafana, Loki or their equivalents, as well as SLOs, incident response, and conducting post-incident reviews.

• Proficient in JavaScript and Node.js, with the ability to review and make architectural decisions concerning control-plane, CLI, and worker services.

• Experience delivering a platform with real users, such as multi-tenant IaaS/PaaS or research computing services.

• Proven ability in people management across different time zones, conducting cross-track reviews, writing architectural decisions, and confidently challenging partners or executives with well-founded arguments.

• Excellent proficiency in written and spoken English.

• Ability to work from UTC through UTC+5:30 to facilitate collaboration with teams in Europe and India.

• Willingness to travel occasionally to partner sites and team events.

🏝️ Benefits

• Enjoy a fully remote work arrangement.

• Travel occasionally to partner sites and team events.

• Join a global, distributed team environment.

• Have the opportunity to work on cutting-edge digital finance, GPU infrastructure, AI, and blockchain platforms.