Customer Reliability Engineer

This is a fully remote position, open to applicants in Europe.

📋 Description

• Take ownership of customer issues from reproduction to diagnosis, resolution, escalation, and closure.

• Diagnose Linux problems related to processes, networking, storage, kernel logs, resource contention, systemd, and journald.

• Explore Kubernetes challenges such as pending or crash-looping pods, node conditions, scheduling failures, and resource limits.

• Troubleshoot GPU malfunctions, including driver and device-plugin issues, XID errors, thermal throttling, cordoning or draining nodes, and multi-node job failures.

• When necessary, escalate issues with the collected evidence.

• Engage in a 24/7 on-call rotation.

• Be the first responder to alerts and customer-reported outages, assess the impact, establish severity, and involve the necessary personnel.

• Keep customers informed during incidents with clear status updates and known unknowns.

• Document incidents and transform insights into runbooks, alerts, or fixes.

• Identify whether issues originate from the provider, Andromeda, or the customer.

• Open and follow up on tickets with compute providers.

• Monitor recurring provider failures and communicate patterns to sourcing decision-makers.

• Write Python or Bash scripts for automating manual checks.

• Develop and enhance cluster and node health monitoring, GPU telemetry, dashboards, and actionable alerts.

• Maintain up-to-date runbooks and customer documentation.

⛳️ Requirements

• Proficient in Linux troubleshooting via the command line, including logs, processes, networking, and disk management.

• Familiarity with Kubernetes, encompassing pods, nodes, deployments, services, scheduling, and failure investigation.

• Capability to write Python or Bash scripts for automating repetitive tasks.

• Strong technical writing skills, with the ability to articulate problems to customers clearly.

• Good judgment in high-pressure situations, including prioritization, escalation, and communication.

• Willingness to participate in a 24/7 on-call rotation.

• Ideal candidates may possess hands-on experience with NVIDIA GPU production, including drivers, CUDA, DCGM, and the Kubernetes device plugin.

• Ideal candidates may have experience in high-performance networking with InfiniBand, RoCE, or NCCL.

• Ideal candidates may have HPC or Slurm experience.

• Ideal candidates may have prior customer-facing technical experience in support engineering, TAM, solutions, or professional services.

• Ideal candidates may have knowledge of tools such as Prometheus, Grafana, Datadog, or similar.

• Ideal candidates may have Infrastructure as Code experience with Terraform, Ansible, or Helm.

• A genuine interest in AI infrastructure and large training jobs is essential.

🏝️ Benefits

• Meaningful equity.

• Healthcare, dental, and vision coverage for employees and their dependents.

• 401(k) plan.

• Unlimited PTO.

• A high-growth environment at a company focused on the AI infrastructure boom.