Staff Storage Platform Engineer – AI Storage

This is a fully remote position, open to applicants in United States.

📋 Description

• Design, develop, and manage the AI storage layer that supports extensive GPU infrastructure.

• Architect scalable storage solutions for both edge and core deployments.

• Establish storage strategies for distributed inference, fine-tuning, and training workloads.

• Create solutions utilizing hyperconverged storage technologies like StorPool, local NVMe, and disaggregated systems such as VAST and Weka.

• Define reference architectures, design principles, reusable patterns, and long-term storage strategies.

• Enhance throughput, latency, data locality, resilience, cost efficiency, and operability for GPU-intensive clusters.

• Benchmark and verify storage performance under realistic AI workloads.

• Implement and sustain CSI drivers, integrating storage with Kubernetes and orchestration systems.

• Incorporate block, object, and shared file storage into the platform.

• Develop multi-tenant, multi-cluster, and multi-site storage environments.

• Create storage architectures for distributed inference platforms such as NVIDIA Dynamo and llm-d.

• Optimize KV-cache persistence, token generation pipelines, and high-concurrency inference workloads.

• Design high-performance data paths using GPU Direct Storage, RDMA/RoCE, and NVMe-oF.

• Lead investigations into storage performance, incident responses, and root-cause analyses.

• Enhance reliability, durability, observability, recovery processes, and operational standards.

• Oversee the comprehensive delivery of storage infrastructure, from architecture and validation to production rollout.

• Drive capacity planning, scaling strategies, lifecycle decisions, and safe production modifications.

• Collaborate with compute, networking, platform, DevOps, operations, deployment teams, vendors, and stakeholders.

• Serve as the primary authority on storage design and mentor engineers.

⛳️ Requirements

• Extensive hands-on experience in designing and operating distributed storage systems for high-performance computing environments.

• Demonstrated experience in crafting storage architectures for large-scale AI inference or training platforms, covering dataset distribution, checkpointing, and KV-cache storage methodologies.

• In-depth understanding of the Linux storage and I/O stack.

• Strong grasp of data access patterns in AI workloads.

• Experience in optimizing storage for GPU-accelerated tasks.

• Familiarity with WEKA Data Platform.

• Knowledge of Kubernetes storage integrations, particularly CSI.

• Experience in managing large-scale storage clusters.

• Profound expertise in designing and operating storage platforms tailored for GPU-heavy environments and distributed AI workloads.

• Acquainted with storage hardware, NVMe devices, storage fabrics, and high-performance data pathways.

• Experience in designing storage observability systems.

• Strong automation capabilities using Python and/or Bash.

• Experience implementing software engineering practices in storage automation and operational tools.

• Proven track record of leading complex technical projects across multiple teams.

• Familiarity with Kubernetes, distributed file systems, object storage, block storage, StorPool, VAST Data, Weka, GPU Direct Storage, RDMA/RoCE, NVMe-oF, and SPDK.

🏝️ Benefits

• Competitive compensation package that reflects your expertise and experience.

• A welcoming work environment defined by friendliness, international diversity, flexibility, and a hybrid-friendly approach.

• Opportunity for exciting career advancement in a rapidly growing scale-up.

• Equal opportunity employer dedicated to diversity and inclusion.