Customer Reliability Engineer

This is a fully remote position, open to applicants in Asia.

📋 Description

• Take ownership of customer issues from the stages of reproduction and diagnosis to resolution, escalation, and closure.

• Analyze and debug Linux issues related to processes, networking, storage, kernel logs, resource contention, systemd, and journald.

• Examine Kubernetes-related problems, including pending or crash-looping pods, node conditions, scheduling failures, and resource constraints.

• Address GPU failures, encompassing driver and device-plugin issues, XID errors, thermal throttling, cordoning or draining nodes, and multi-node job failures.

• Escalate issues with the evidence that has already been compiled.

• Participate in a 24/7 on-call rotation.

• Be the first responder to alerts and customer-reported outages, evaluate impact, determine severity, and engage the necessary personnel.

• Keep customers informed throughout incidents.

• Document incidents and transform findings into runbooks, alerts, or solutions.

• Assess whether faults originate from the provider side, Andromeda side, or customer side.

• Initiate and follow up on tickets with compute providers.

• Monitor recurring provider failures and communicate patterns to sourcing decision-makers.

• Create automation scripts in Python or Bash.

• Develop and enhance monitoring for cluster and node health, GPU telemetry, dashboards, and alerts.

• Keep runbooks and customer documentation up to date.

⛳️ Requirements

• Proven ability in Linux troubleshooting through the command line, including logs, processes, networking, and disk.

• Familiarity with Kubernetes concepts such as pods, nodes, deployments, services, scheduling, and failure investigation.

• Proficiency in writing Python or Bash scripts for task automation.

• Strong technical writing skills and the ability to clearly communicate issues to frustrated customers.

• Sound judgment when working under pressure.

• Willingness to engage in a 24/7 on-call rotation.

• Experience with NVIDIA GPUs, including familiarity with drivers, CUDA, DCGM, and Kubernetes device plugins, may be beneficial.

• High-performance networking experience, particularly with InfiniBand, RoCE, or NCCL, may be advantageous.

• Experience with HPC or Slurm is a plus.

• A background in a customer-facing technical role may be advantageous.

• Knowledge of monitoring tools like Prometheus, Grafana, Datadog, or similar platforms may be beneficial.

• Experience with infrastructure as code tools such as Terraform, Ansible, or Helm may be advantageous.

• A genuine interest in AI infrastructure may be an added advantage.

🏝️ Benefits

• Meaningful equity.

• Healthcare coverage for you and your dependents.

• Dental coverage for you and your dependents.

• Vision coverage for you and your dependents.

• 401(k) plan.

• Unlimited PTO.

• A high-growth environment at a company positioned at the forefront of the AI infrastructure boom.