Senior Production Engineer – DGX Cloud

This is a fully remote position, open to applicants in Switzerland.

📋 Description

• Develop and manage production software, automation, and tools for control plane services, model deployments, and inference and agentic tasks within DGX Cloud environments.

• Enhance the reliability of inference and agentic platforms and services, such as NVIDIA Cloud Functions, SGLang- and vLLM-based endpoints, and inference services utilizing NVIDIA Dynamo, through health validation, safer rollouts, observability, and recovery.

• Boost endpoint availability, inference routing, capacity management, and service health to ensure consistent performance as workloads and demand fluctuate.

• Employ infrastructure as code and GitOps methodologies to deploy, configure, validate, upgrade, and recover services uniformly across environments.

• Create workflows for service enablement, model releases, handoff procedures, deprecation, and ongoing operations; automate repetitive manual tasks with dependable automation.

• Establish and monitor SLIs and SLOs for inference and control plane services, focusing on availability and latency, utilize error budgets to steer reliability enhancements, and ensure production health is visible to partner teams.

• Engage in on-call duties and incident response, troubleshoot failures across routing, model runtimes, software, and infrastructure, and transform recurring issues into automated solutions and lasting fixes.

• Collaborate with model, platform, storage, networking, security, and GPU infrastructure teams to design and manage services securely at scale.

⛳️ Requirements

• Over 8 years of experience in building or managing production services and large-scale distributed systems, including practical experience in automation.

• Proficient programming abilities in Python, Go, or a similar language, with experience in developing production operation tools.

• Familiarity with infrastructure as code, configuration management, or GitOps, and experience in creating automation for repeatable service deployments and modifications.

• In-depth knowledge of Linux, Kubernetes, containers, cloud infrastructure, distributed systems, and networking essentials; capable of diagnosing production failures.

• Comprehension of Production Engineering principles, including SLIs, SLOs, error budgets, incident response, and minimizing operational toil.

• Experience in instrumenting services and utilizing metrics, logs, and traces to analyze system behavior and enhance reliability.

• Strong technical communication skills and the ability to collaborate across engineering teams.

• BS/MS in Computer Science or equivalent practical experience.

• Knowledge of vLLM, SGLang, PyTorch, TensorRT-LLM, NVIDIA Dynamo, CUDA, or NCCL, along with GPU performance analysis.

• Experience in building Kubernetes operators, controllers, workload orchestration services, fleet management systems, or self-healing automation.

• Proficiency with Terraform, Argo CD, CI/CD, policy validation, or secure deployment and rollback systems.

• Experience in developing AI tools and agents.

• Familiarity with production AI inference or agentic workloads, including troubleshooting issues across models, runtimes, Kubernetes, and hardware.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health and wellness benefits.

• Opportunities for professional development and continuous learning.

• Flexible working hours and remote work options.

• Engaging work environment with a focus on innovation.