Senior Engineer

This is a fully remote position, open to applicants in California, +1 more state.

📋 Description

• Oversee NVIDIA Cloud Partner Day 2 operational readiness initiatives post-initial deployment and activation.

• Partner with NVIDIA Cloud Partners to create systems, procedures, automation, and operational methodologies for optimized infrastructure.

• Develop ongoing validation for GPU, CPU, storage, and network health within extensive AI clusters.

• Set up telemetry, monitoring, alerting, dashboards, and operational indicators across compute, GPU, networking, storage, Kubernetes, and AI workloads.

• Create automated workflows to detect, isolate, drain, repair, validate, and restore non-functional infrastructure back to service.

• Oversee the GPU fleet lifecycle, encompassing drivers, firmware, Kubernetes nodes, OS updates, configuration management, upgrades, and configuration drift.

• Convert NVIDIA NCP requirements and reference architectures into production operational practices, validation standards, runbooks, automation, and measurable benchmarks.

• Define health indicators, SLOs, metrics, acceptance standards, and infrastructure readiness validation.

• Develop reusable tools, automation, implementation guides, runbooks, playbooks, and reference implementations throughout NCP environments.

⛳️ Requirements

• BS, MS, or Ph.D. in Computer Science, Computer/Electrical Engineering, or a related technical discipline, or equivalent experience.

• Over 8 years of experience in infrastructure engineering, Site Reliability Engineering, DevOps, cloud platform engineering, systems engineering, or comparable roles in large-scale production settings.

• Extensive experience in managing Linux-based distributed systems and cloud infrastructure in a production environment.

• Profound knowledge of Kubernetes, containers, cluster scheduling, and the operational lifecycle of large multi-node environments.

• Strong grasp of production observability, including metrics, logging, alerting, dashboards, health checks, and operations driven by service level agreements.

• Experience in creating automation for infrastructure lifecycle management, failure detection, remediation, upgrades, and configuration management.

• Solid networking principles and experience troubleshooting complex distributed systems across compute, network, and storage layers.

• Programming and automation skills using Python, Go, shell scripting, or similar languages.

• Experience managing large GPU or accelerated computing infrastructure that supports AI training and inference workloads.

• Familiarity with NVIDIA technologies such as DGX/HGX systems, CUDA, NVLink/NVSwitch, NVIDIA networking, InfiniBand, RoCE, GPU Operator, Network Operator, or related NVIDIA infrastructure software.

• Proven track record of collaborating with NVIDIA Cloud Partners, hyperscale cloud providers, managed AI clouds, or extensive service-provider infrastructure while operating SLOs for large-scale compute infrastructure and utilizing operational data to enhance availability, performance, and fleet efficiency.

• In-depth knowledge of infrastructure observability tools including Prometheus, Grafana, OpenTelemetry, Alertmanager, and scalable telemetry pipelines.

• Understanding of failure modes associated with large distributed AI workloads and the necessary infrastructure features to consistently support prolonged training and production inference.

🏝️ Benefits

• Equity

• Benefits