AIOps Engineer, AI Infrastructure

This is a fully remote position, open to applicants in California.

📋 Description

• Verify system health prior to validation by checking node readiness, scheduler health, GPU availability, and essential cluster services.

• Ensure stable operation during validation by addressing issues related to Linux, SLURM, GPU, network, and storage, while collaborating with appropriate teams to restore optimal system performance.

• Distinguish between infrastructure faults and workload or test-related problems, transforming insights into actionable next steps, remediation strategies, and operational guidance.

• Enhance resolution of recurring issues through improved diagnostics, runbooks, alerts, and targeted automation that minimizes manual efforts and enables more efficient team responses.

⛳️ Requirements

• A minimum of 4 years of experience in site reliability engineering, infrastructure engineering, systems engineering, or production operations within large-scale Linux environments.

• Proficient in Linux administration and troubleshooting across services, processes, networking, storage, resource contention, and system performance.

• Experience in operating or troubleshooting bare-metal, HPC, or GPU clusters, including SLURM administration, with a solid understanding of cluster health and workload execution.

• Competency in Python or Bash scripting, along with familiarity with tools like Ansible or similar.

• Experience with monitoring, telemetry, logs, dashboards, alerts, or diagnostic tools such as Prometheus, Grafana, Alertmanager, or equivalent platforms.

• Proven experience in troubleshooting and remediation, including failure investigation, root cause analysis, and restoring system health.

• A bachelor’s degree in computer science, computer engineering, electrical engineering, or a related field, or equivalent practical experience.

• Preferred: experience with GPU telemetry and SLURM metrics.

• Preferred: familiarity with bare-metal cluster management platforms.

• Preferred: practical knowledge of bare-metal provisioning, InfiniBand, RoCE, NVLink or NVSwitch, and parallel storage failure modes.

• Preferred: experience in supporting early-life cluster operations, system validation, or customer deployment environments.

🏝️ Benefits

• Consideration for equal opportunity employment.

• Opportunities for growth, learning, ownership, and impactful contributions.

• Freedom to think creatively and space to develop professionally.