Research Engineer – Pre-training

This is a fully remote position, open to applicants in United States, +1 more country.

📋 Description

• Implement and enhance model-parallel distributed pretraining utilizing data, pipeline, and tensor parallelism.

• Train large-scale models on diverse GPUs using low-bandwidth, high-latency internet connections.

• Apply methods to minimize communication overhead while ensuring model convergence is preserved.

• Develop elasticity and fault tolerance for node churn, which includes checkpointing, state synchronization, and participant recovery.

• Create run instrumentation and monitoring systems to assess throughput, identify bottlenecks, and evaluate model quality across numerous devices.

• Advance Protocol Learning from an 8B model run to a frontier scale.

⛳️ Requirements

• Practical experience in training models across multiple devices using PyTorch with FSDP, DeepSpeed, Megatron, or a similar framework.

• Knowledge of data, tensor, and pipeline parallelism.

• Proficiency in production-quality Python programming.

• Familiarity with concurrency, managing failures, and profiling prior to optimization.

• Demonstrated experience with shipped systems, research code, open-source contributions, or significant personal projects.

• Professional-level proficiency in English, both written and spoken.

• Ability to work comfortably across different time zones.

• Preferred: experience in training or serving large language models such as Nemotron, Qwen, or OLMo.

• Preferred: experience with P2P networking and NAT traversal.

• Preferred: experience in post-training and reinforcement learning.

• Preferred: experience with inference and serving systems.

• Preferred: experience working at proprietary, open-weight, or open-source AI labs.

🏝️ Benefits

• Significant equity ownership alongside a competitive base salary.

• Flexible remote work options.

• Full visa sponsorship and relocation assistance to Australia or the US for outstanding candidates.

• Opportunity to engage with open, frontier-scale research challenges.

• Collaborate with a globally distributed team.