Senior Solutions Architect – AI Factory Observability and Visualization

This is a fully remote position, open to applicants in California, +2 more states.

📋 Description

• Execute validation tools, microbenchmarks, and workloads related to AI factory as provided by the team.

• Acquire an in-depth understanding of the entire system lifecycle.

• Define what constitutes a "healthy" system across the entire stack.

• Create and enhance the telemetry interface across hardware, fabric, and workloads.

• Act as the expert in observability, identifying areas lacking visibility.

• Design automation (using Python and Shell) for the collection, transformation, and presentation of system and network data.

• Suggest enhancements to system visibility, data sources, and reporting mechanisms.

• Collaborate with teams in hardware, software, networking, data centers, and product development.

⛳️ Requirements

• A Bachelor's degree or equivalent experience in Computer Science, Mathematics, Engineering, Physics, or a related discipline.

• Over 6 years of experience managing Linux-based systems in high-performance computing (HPC), distributed systems, or extensive AI/ML environments.

• Practical experience with the architecture of multi-GPU and/or multi-node clusters.

• A strong understanding of how HPC and AI factory systems integrate end-to-end.

• Proficient in Python and Shell/Bash for scripting, automation, and development of tools.

• Hands-on experience with observability systems such as Prometheus, Grafana, Loki, or similar platforms.

• Experience in transforming metrics, logs, and traces into actionable insights for complex distributed systems.

• Knowledge of GPU and fabric telemetry, including tools like DCGM, NVLink, and InfiniBand/Ethernet fabric counters.

• Excellent communication skills and the capability to work effectively within cross-functional teams.

🏝️ Benefits

• Eligibility for equity opportunities.

• Comprehensive health insurance.

• Opportunities for professional development.