
GPU Infrastructure Engineer
Posted 2 days ago

Posted 2 days ago
This is a fully remote position, open to applicants in Canada.
• Expand PolarGrid's inference network from a limited number of sites to a widely distributed array of GPU clusters.
• Take ownership of the onboarding process for new sites and nodes, overseeing everything from provisioning and burn-in to validation and phased production rollout.
• Determine when a site is ready to handle customer traffic and ensure new locations integrate into the network without compromising latency or reliability.
• Operate the global routing layer that directs requests to the most optimal serving location.
• Utilize network latency, capacity, and health indicators to adjust routing in response to live traffic.
• Establish and monitor availability objectives throughout the network.
• Develop health checks and automation systems to identify and reroute around malfunctioning hardware and models.
• Ensure rapid recovery and consistently test failure management protocols.
• Create observability, release, and network systems to efficiently manage a large-scale distributed GPU fleet.
• Implement comprehensive end-to-end tracing and latency alerting mechanisms.
• Oversee safe and controlled staged rollouts.
• Maintain network configuration as code.
• Enforce security measures and DDoS protection at the network's edge.
• Proven experience operating distributed systems across multiple locations or regions with high availability requirements.
• Strong understanding of networking principles: BGP, DNS, TCP/TLS, and load balancing.
• Practical experience managing bare-metal GPU fleets, including provisioning, burn-in, firmware updates, and node lifecycle management.
• Familiarity with InfiniBand or RDMA networking.
• Experience with LLM inference engines such as vLLM, SGLang, or TensorRT-LLM, focusing on multi-node serving, cold starts, and TTFT.
• Advanced observability skills, particularly in monitoring p95/p99 latency and gray failures, beyond mere uptime metrics.
• Background in designing failover strategies, graceful degradation, and automated recovery processes.
• Strong automation and infrastructure-as-code perspective, including CI/CD practices for infrastructure.
• Capability to plan capacity amidst constrained and uneven hardware availability.
• Experience in developing internal tools or control planes, preferably using Go or Rust.
• Knowledge of DDoS mitigation techniques.
• Experience managing DNS infrastructure at scale.
• Understanding of hardware security measures: TPM, measured boot, or confidential computing.
• Experience with real-time streaming inference, particularly in voice and audio applications.
• Ability to operate remote datacenter sites with on-site technicians.
• Experience with certificate automation across multiple domains.
• Proficiency in deterministic simulation testing.
• Competitive salary and performance-based incentives.
• Comprehensive health and wellness benefits.
• Opportunities for professional development and career advancement.
• Flexible work arrangements and a supportive work environment.
Qompyl
Curtiss-Wright Corporation
Health Care Service Corporation
Get handpicked remote jobs straight to your inbox weekly.