Senior/Staff Software Engineer, Inference, Compute Infrastructure Engineering

This is a fully remote position, open to applicants in India.

📋 Description

• Develop the provisioning state machine that models the entire physical-host lifecycle, from discovery and inference bring-up to GPU driver/CUDA installation, health validation, and decommissioning/RMA.

• Create declarative self-service APIs and a control plane for requesting, scaling, and dismantling inference clusters.

• Automate self-healing processes by identifying degraded or malfunctioning nodes, safely draining them, initiating repair or replacement, and restoring healthy capacity.

• Take ownership of pipeline reliability through idempotency, retries, rollback, and drift detection.

• Collaborate with the inference/ML platform team to integrate cluster topology, interconnect, and scheduling constraints as platform abstractions.

• Implement strong typing, automated testing, code reviews, versioning, and CI/CD practices in infrastructure code.

• Develop Kubernetes-native controllers, reconciliation loops, self-service API/CLI interfaces, and event-driven health, remediation, and utilization systems for an optimized GPU inference fleet.

• Manage and support the software in a production environment.

⛳️ Requirements

• Solid software engineering experience in Go, Python, Rust, or a similar language — you write and test real software as part of your profession.

• Familiarity with durable workflow orchestration tools such as Temporal, Cadence, or equivalents to execute long-lived, manifest-driven workflows that can withstand failures and resume mid-execution.

• Experience in building software control planes or orchestration systems that represent state and reconcile it over time (e.g., Kubernetes controllers/operators, custom reconciliation loops, workflow engines).

• Background in event-driven systems — designing and implementing software using message queues, event streams, or pub/sub mechanisms (e.g., Kafka, NATS, SQS) rather than relying on polling or cron-driven scripts.

• A product-oriented mindset; experience in creating internal platforms or APIs that are utilized by other engineering teams and a focus on enhancing developer experience.

• Nice to have: Familiarity with bare-metal provisioning (PXE/iPXE, Redfish/IPMI, BMC), networking principles (VLANs, BGP, fabric design), or GPU/accelerator infrastructure.

• Nice to have: Experience with GPU cluster software stacks (NCCL, CUDA, InfiniBand/RoCE).

• Nice to have: Previous experience at a hyperscaler, GPU cloud, or datacenter-scale infrastructure organization.

• Nice to have: Systems programming experience in Rust or Go.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health insurance and wellness programs.

• Opportunities for professional development and career advancement.

• Flexible work hours and remote work options.

• Collaborative and innovative work environment.