Senior SRE, Automation Engineer – Customer Facing

This is a fully remote position, open to applicants in California, +1 more state.

📋 Description

• Take full responsibility for the reliability of the customer-facing GPU cloud service, including aspects such as availability, job completion, provisioning latency, and tenant experience.

• Manage production Kubernetes clusters optimized for GPU workloads, ranging from 100 to 10,000 GPUs.

• Oversee the Nvidia GPU operator, device plugin, MIG configuration, GPU time-slicing, and policies for multi-tenant GPU allocation.

• Implement scheduling that is aware of topology, considering GPU locality, NVLink domains, and network rail affinity.

• Handle tenant onboarding, quota management, enforcement of isolation, and offboarding/reclamation processes.

• Automate the provisioning of Bare-Metal-as-a-Service, tenant handoffs, lifecycle management, and reclamation tasks.

• Define and publish SLIs, SLOs, and SLAs that are customer-facing.

• Lead the detection of incidents, their remediation, escalation processes, customer-facing status updates, and post-incident reviews.

• Develop and maintain tenant-aware monitoring and observability tools using Prometheus, Grafana, Alertmanager, and PagerDuty.

• Automate the detection of GPU node failures, as well as actions such as drain/cordon/taint and rescheduling of workloads.

• Develop infrastructure as code, leveraging tools such as Terraform, Helm, and GitOps across GPU clusters.

• Create and maintain service documentation, tenant runbooks, capacity plans, and support tiering structures.

• Collaborate with customer success and support teams to translate customer-reported issues into systemic enhancements.

• Construct self-service observability features for customers regarding their status, quotas, and job health.

• Ensure the control plane is secure for automated remediation through the use of CRDs and executable runbooks.

• Achieve published and met SLAs, automate GPU fault recovery, facilitate external-tenant BMaaS onboarding, and reduce MTTD/MTTR, while providing tenant self-service observability.

⛳️ Requirements

• Minimum of 5 years' experience in SRE or cloud operations, with at least 2 years focused on managing GPU workloads at scale.

• Profound understanding of Kubernetes operations and GPU workload management, including Nvidia GPU operator, device plugin, MIG, time-slicing, and GPU scheduling.

• Experience in topology-aware scheduling and management of GPU-specific resources.

• Practical experience in building multi-tenant cloud platforms that offer robust isolation guarantees.

• Customer-facing cloud service experience, with skills in defining and managing SLAs/SLOs and handling tenant incidents and communications.

• Familiarity with bare-metal server provisioning and lifecycle automation using Ironic, MAAS, or custom solutions.

• Proficient in Terraform, Helm, and GitOps workflows, with experience in ArgoCD or Flux.

• Strong background in SRE, including familiarity with SLI/SLO/SLA frameworks, error budgets, incident management, and capacity planning.

• Experience with Prometheus, Grafana, and large-scale alerting systems.

• Strong programming skills in Go or Python for the development of automation and operators.

• Aptitude for AIOps, treating the control plane as an execution surface for automated remediation.

• Embrace a runbook-as-code mindset, where SRE playbooks are designed to be executable by the platform.

🏝️ Benefits

• Competitive salary and comprehensive benefits package.

• Opportunity for professional growth and development.

• Flexible work environment with remote work options.

• Engaging and innovative team culture.