Senior Systems Software Engineer – Data Center Performance

This is a fully remote position, open to applicants in Poland, +2 more countries.

📋 Description

• Characterize the performance of nodes, racks, and clusters utilizing microbenchmarks and representative applications.

• Evaluate throughput, latency, and scaling efficiency to uncover compute, memory, communication, and storage bottlenecks.

• Diagnose Linux performance issues related to CPU scheduling, NUMA locality, CPU and memory affinity, interrupt handling, and resource contention.

• Analyze the effects of Linux performance challenges on GPU utilization and application efficiency.

• Investigate distributed workload scaling, collective communication, synchronization delays, slow nodes, and the interplay between computation and communication.

• Assess how the positioning of GPUs and NICs, the structure of interconnects, and network traffic affect overall performance.

• Integrate application profiles, system traces, hardware counters, and cluster telemetry to pinpoint root causes.

• Create focused reproductions and work collaboratively with hardware, firmware, driver, library, and application teams to implement and validate solutions.

• Develop and maintain Python tools for the automated execution of benchmarks, data collection, and performance analysis.

• Establish repeatable baselines, evaluate run-to-run variability, and detect regressions across different hardware setups and software versions.

• Define and execute performance validation for new platforms and cluster deployments.

• Identify nodes, links, or storage paths that are underperforming and confirm that corrective measures restore workload performance.

⛳️ Requirements

• Extensive hands-on experience in Linux performance analysis, focusing on processes and threads, CPU and memory architecture, NUMA, I/O, and networking.

• Experience in performance investigation within multi-node HPC or GPU-accelerated systems.

• In-depth expertise in at least one of the following areas: GPU application performance, Linux systems performance, high-speed networking and collective communication, parallel storage, or HPC performance tooling.

• Proficient in Python programming.

• Experience in developing and debugging performance-sensitive code using C, C++, and CUDA.

• Capability to design controlled experiments, interpret profiling and counter data, and achieve reproducible improvements.

• Ability to independently lead technical initiatives across hardware and software domains and effectively communicate findings.

• BS or MS in Computer Science, Engineering, Mathematics, or Physics, or equivalent professional experience.

• Over 8 years of relevant experience.

• Familiarity with Nsight Systems, Nsight Compute, Linux perf, or eBPF-based tools.

• Experience with RDMA and MPI or NCCL.

• Knowledge of analyzing PCIe, NVLink, InfiniBand, or Ethernet interconnect performance.

• Experience with Slurm, container runtimes, and cgroups.

• Proven track record of enhancing distributed training, inference, or scientific applications through strong- and weak-scaling studies.

• Experience in building automated performance regression frameworks utilizing tools such as ReFrame.

🏝️ Benefits

• Competitive salary and comprehensive benefits package.

• Opportunities for professional development and continuous learning.

• Flexible working hours and a supportive work environment.

• Access to cutting-edge technology and tools.