Senior Cloud Infrastructure and Reliability Solutions Architect

This is a fully remote position, open to applicants in France, +3 more countries.

📋 Description

• Take responsibility for Day 2 production stability and reliability at fleet scale across EMEA NVIDIA Cloud Partner estates.

• Establish, evaluate, and enhance MTBI and job goodput on multi-thousand-GPU clusters while managing platform error budgets.

• Develop and maintain automated workflows for fault detection and remediation concerning GPU, node, fabric, and storage failures.

• Streamline processes for triage, node draining, cordoning, resetting, and returning nodes to service.

• Execute preventive maintenance and proactive firmware and field-notice rollout initiatives, encompassing risk assessment, staged implementation, maintenance-window planning, and verification.

• Manage observability and reliability tools for GPU fleets, incorporating Prometheus, Grafana, Loki, DCGM, XID diagnostics, GPUd, and NVSentinel into partner platforms.

• Lead incident response efforts and conduct post-incident reviews in collaboration with NVEX, TAMs, and partner operations teams.

• Facilitate root-cause analyses, track corrective actions to completion, and transform recurring incidents into automation, health checks, or runbooks.

• Oversee Kubernetes node lifecycle and reliability engineering across upstream Kubernetes, KubeVirt, Slurm, and GPU-aware schedulers.

• Address node health policies, manage upgrade and reboot orchestration, and ensure safe workload draining on GPU-accelerated nodes.

⛳️ Requirements

• BS/MS/PhD in Computer Science, Electrical/Computer Engineering, Physics, Mathematics, or related fields, or equivalent practical experience.

• Over 8 years of experience in SRE, production engineering, or large-scale infrastructure operations roles.

• Proven track record of overseeing production reliability at scale, including SLO and error-budget establishment, on-call and incident command, blameless post-incident reviews, and measurable enhancements in availability or mean time between interruptions.

• Practical experience with operating HPC/AI clusters and NVIDIA GPU-accelerated infrastructure, including driver and CUDA toolkit lifecycle management, firmware oversight, workload profiling, and troubleshooting across CPUs, GPUs, and high-speed interconnects.

• Extensive knowledge of running Kubernetes in GPU-accelerated and HPC settings, covering node lifecycle, upgrades, draining, scheduler behavior, Slurm, and mixed bare-metal/virtualized multi-tenant environments.

• Proficient in Python and Bash, Infrastructure-as-Code, configuration management tools such as Ansible and Terraform, GitOps-based fleet lifecycle and upgrade management, as well as Prometheus, Grafana, and Loki.

• In-depth understanding of Linux, including RedHat and Ubuntu, OS-level tuning and security, data center networking fundamentals, and parallel/enterprise storage solutions including Lustre, GPFS, ZFS, and XFS.

• Experience in defining and executing proactive firmware, microcode, or field-notice rollout campaigns across a substantial production fleet.

• Familiarity with DCGM, XID diagnostics, GPUd, NVSentinel, node-level health agents, and fleet-wide reliability intelligence.

• Experience with NVIDIA GPU and Network Operators and NVIDIA Base Command Manager.

• Practical expertise in AI-native scheduling and inference frameworks on Kubernetes, such as KAI, Grove, Dynamo, and NVIDIA Cloud Functions.

• Specialized knowledge in RDMA-based fabrics like InfiniBand or RoCE; exposure to Cumulus Linux, SONiC, Spectrum-X, DPU/DOCA, and NVLink/NVSwitch partition operations is a plus.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Generous paid time off and flexible working arrangements.

• Opportunities for professional development and continuous learning.

• Access to cutting-edge technology and innovative projects.