Research Scientist / Engineer – Training Infrastructure

This is a fully remote position, open to applicants in Europe.

📋 Description

• Design, develop, and enhance efficient distributed training systems for models utilizing thousands of GPUs.

• Investigate and implement cutting-edge parallelization techniques, such as FSDP, Tensor Parallel, Pipeline Parallel, and Expert Parallel.

• Create monitoring, visualization, and debugging tools for extensive training processes.

• Enhance training stability, convergence, and resource efficiency across large clusters.

• Familiarize oneself with the existing training stack and troubleshoot stability and utilization challenges on a large scale.

• Achieve a parallelization or stability enhancement that significantly impacts a real training run.

• Develop monitoring and tools to ensure large training runs are reliable and efficient.

⛳️ Requirements

• Extensive experience in distributed PyTorch training and parallelism within foundation-model training.

• Profound knowledge of GPU clusters, networking, and storage systems.

• Familiarity with communication libraries (NCCL, MPI) and optimization of distributed systems.

• Strong skills in Linux systems administration and scripting (preferred).

• Experience managing training runs with over 100 GPUs (preferred).

• Knowledge of containerization, orchestration, and cloud infrastructure (preferred).

• Proficiency in FSDP and multi-node training.

• Capability to work remotely within the EU.

🏝️ Benefits

• Equal opportunity employer.

• Participation in a voluntary diversity and inclusion survey; opting out will not impact the job application.

• Remote work flexibility.