Senior Systems Software Engineer – HPC and AI Storage

This is a fully remote position, open to applicants in Poland, +2 more countries.

📋 Description

• Design and implement storage systems for extensive HPC and AI clusters, utilizing Lustre, NFS, and ensuring their integration with compute nodes, networks, and storage hardware.

• Analyze performance and reliability challenges across the I/O stack, overseeing application interactions across filesystem clients, Linux kernel pathways, network protocols, servers, and storage devices.

• Collaborate with NVIDIA engineering teams, storage partners, and open-source maintainers to develop software fixes and performance enhancements through review, testing, and deployment.

• Characterize application I/O patterns, including training data access, checkpointing and restarting, metadata-heavy operations, and concurrent workloads; convert insights into software, configuration, and design enhancements.

• Create reproducible benchmarks, diagnostic tools, and regression tests that assess efficiency, latency, metadata performance, scalability, and contention behavior.

• Oversee storage qualification and commissioning, ensuring functionality, data integrity, failure recovery, and sustained performance at cluster scale.

• Establish and execute storage placement, caching, and data lifecycle policies based on workload demands, storage capacity, and operational data.

⛳️ Requirements

• Extensive engineering experience with Lustre, NFS, or similar shared storage systems, including in-depth debugging at the implementation level in at least one stack and experience integrating storage into Linux clusters.

• Proficient in C programming and Linux debugging, with a background in reading, modifying, and diagnosing kernel or user-space storage software.

• Knowledgeable about filesystem and distributed-system behavior, encompassing metadata operations, caching and consistency, locking, concurrency, client/server communication, and recovery mechanisms.

• Proven experience in analyzing application access patterns and optimizing filesystem or storage-stack behavior to achieve performance enhancements.

• Capable of managing a complex performance or correctness issue from initial symptoms through to reproduction, root-cause analysis, implementation, and validation.

• Skilled in Python and shell scripting for developing maintainable diagnostic, test automation, and analysis tools.

• Demonstrated ability to independently lead technical initiatives, influence design decisions across teams, and clearly communicate findings and trade-offs.

• Bachelor’s or Master’s degree in Computer Science, Engineering, Mathematics, Physics, or equivalent practical experience, with a minimum of 12 years of relevant experience.

• Contributions to Lustre, Linux filesystems, NFS implementations, or other storage projects, including features, performance enhancements, bug fixes, or technical reviews.

• Expertise in Lustre metadata and client I/O, Linux VFS and page cache, NFS protocol and client/server internals, or distributed locking and recovery mechanisms.

• Familiarity with hierarchical storage management, policy engines, or migration between filesystems and object storage, including data recall and recovery from interrupted transfers.

• Experience with RDMA storage transports, GPUDirect Storage, or optimizing data loading and checkpointing for GPU workloads.

• Knowledge of storage security and multi-tenancy, including authentication, identity mapping, permissions, and encryption practices.

🏝️ Benefits

• Competitive salary and performance-based bonuses.

• Comprehensive health, dental, and vision insurance.

• Generous retirement savings plan with employer matching.

• Opportunities for professional development and continuing education.

• Flexible work hours and the option for remote work.

• A collaborative and innovative work environment.