Research Scientist – Pre-training

This is a fully remote position, open to applicants in United States, +2 more locations.

📋 Description

• Develop a training system that scales Protocol Learning from an 8B run to frontier levels.

• Execute and enhance model-parallel distributed pretraining across diverse GPUs.

• Implement data, pipeline, and tensor parallelism over low-bandwidth, high-latency connections.

• Minimize communication overhead while ensuring model convergence.

• Establish elasticity and fault tolerance to ensure runs endure node churn.

• Implement reliable checkpointing, state synchronization, and recovery as participants join and exit.

• Create monitoring systems to display throughput, identify bottlenecks, and assess model quality across numerous devices.

• Train and serve large models on varied hardware located in different physical regions using standard internet connections.

⛳️ Requirements

• Practical experience in distributed training is essential: have trained models across multiple devices in PyTorch leveraging FSDP, DeepSpeed, Megatron, or a custom solution.

• Knowledge of data, tensor, and pipeline parallelism is required.

• Proficiency in production-quality Python is necessary.

• Experience in handling concurrency and failures is preferred.

• Familiarity with profiling prior to optimization is required.

• Proven track record of execution via deployed systems, research code, open-source projects, or significant personal initiatives.

• Alignment with the mission of Protocol Learning is essential.

• Professional-level proficiency in English, both written and spoken, is required.

• Nice to have: experience in training or serving large language models such as Nemotron, Qwen, or OLMo.

• Nice to have: familiarity with P2P networking and NAT traversal.

• Nice to have: experience in post-training and reinforcement learning.

• Nice to have: background in inference and serving systems.

• Nice to have: experience in proprietary, open-weight, or open-source AI labs.

🏝️ Benefits

• Significant ownership/equity for key technical contributors.

• Competitive base salary.

• Flexible work environment.

• Full visa sponsorship and relocation assistance to either Australia or the US for exceptional candidates.

• Opportunity to tackle open, unsolved research challenges.

• Remote-first culture.

• Globally distributed team.