Senior Storage Engineer

This is a fully remote position, open to applicants in United States, +1 more country.

📋 Description

• Take responsibility for the complete storage architecture of the fleet, which encompasses durable object storage, high-throughput storage per pod, node-local NVMe cache, and replication and placement across the United States and Europe.

• Design and construct the model distribution pathway, incorporating chunking, deduplication, prefetching, pinning, and pod working-set warmup.

• Profile and minimize model loading latency from disk to GPU memory.

• Optimize the kernel, file system, NVMe-oF or RDMA transport, GPUDirect Storage, and inference-engine loader.

• Choose, benchmark, and manage parallel/distributed file systems, object storage, NVMe drives, enclosures, and management software.

• Validate storage selections on company hardware prior to pod deployment.

• Manage capacity planning, monitoring, alerting, drive health, failure prediction, rebuilds, backups, recovery, and storage-layer on-call responsibilities.

• Develop documented, automated storage provisioning and validation playbooks for new 1 MW pods.

• Collaborate with inference engine, platform and hardware engineering teams, storage and NVMe vendors, along with site deployment teams.

• Leverage LLM-based agents to automate benchmarks, runbooks, triage, and tooling.

• Oversee data residency and operational storage security, which includes encryption at rest, access control, retention, and in-country placement.

⛳️ Requirements

• Over 8 years of experience in storage or systems engineering, with several years dedicated to designing and operating storage in a company with its own hardware at scale.

• Proven experience in constructing storage systems from design to production levels reaching petabytes.

• Extensive, up-to-date, hands-on expertise in Linux storage, including the block layer, io_uring, NVMe queues, XFS, ext4, ZFS, page cache, and I/O analysis tools.

• Production experience with at least one parallel or distributed file system, such as Lustre, WEKA, VAST, GPFS, or Ceph.

• Familiarity with S3-compatible object storage at a petabyte scale.

• Strong grasp of NVMe, NVMe-oF, RDMA transports, and storage networking.

• Proficient in a systems programming language: Go, Rust, or C/C++.

• Proficient in Python.

• Experienced in developing tooling, benchmarks, and automation.

• Regularly utilize AI for scripting, benchmarking, analysis, documentation, and debugging.

• Capability to take ownership of outcomes in a fast-paced, ambiguous environment and make decisions with limited information.

• Willingness to travel to sites in the US and Europe as required.

🏝️ Benefits

• Meaningful stock options.

• Remote-first work arrangement.

• Flexible schedule outside of core collaboration hours.

• Generous paid time off, including vacation, sick days, and public holidays.

• Paid maternity, paternity, and caregiver leave.

• Twice-yearly company retreats.

• In-person meetings held twice a year.