AI Infrastructure Engineer

This is a fully remote position, open to applicants in Texas.

📋 Description

• Verify system health prior to validation by assessing node readiness, scheduler functionality, GPU availability, and essential cluster services.

• Facilitate stable operations during validation by diagnosing issues related to Linux, SLURM, GPU, network, and storage.

• Collaborate with relevant teams to restore optimal system functionality.

• Distinguish between infrastructure malfunctions and workload or test-related problems.

• Translate findings into actionable next steps, remediation strategies, and operational guidance.

• Enhance the management of recurring problems through improved diagnostics, runbooks, alerts, and targeted automation.

• Minimize manual workload and assist teams in responding more effectively.

⛳️ Requirements

• A minimum of 4 years of experience in site reliability engineering, infrastructure engineering, systems engineering, or production operations within large-scale Linux environments.

• Proficient Linux administration and troubleshooting capabilities across services, processes, networking, storage, resource contention, and system performance.

• Experience in operating or troubleshooting bare-metal, HPC, or GPU clusters, including SLURM administration and a solid understanding of cluster health and workload execution.

• Practical skills in Python or Bash scripting and familiarity with Ansible or similar tools for operational tasks.

• Experience with monitoring, telemetry, logs, dashboards, alerts, or diagnostic tools such as Prometheus, Grafana, Alertmanager, or comparable platforms.

• Proven troubleshooting and remediation expertise, including the ability to investigate failures, identify root causes, and restore healthy system operations.

• A bachelor’s degree in computer science, computer engineering, electrical engineering, or a related field, or equivalent practical experience.

• Experience with GPU telemetry and SLURM metrics.

• Familiarity with bare-metal cluster management platforms.

• Practical knowledge of bare-metal provisioning, InfiniBand, RoCE, NVLink or NVSwitch, and parallel storage failure modes.

• Experience supporting early-life cluster operations, system validation, or customer deployment environments.

🏝️ Benefits

• Equal opportunity employer.

• Opportunity to grow, take ownership, and create real impact from day one.