This is a fully remote position, open to applicants in California.

📋 Description

• Develop and enhance the pretraining code within prime-rl, progressing small experiments into comprehensive training sessions.

• Assess modifications in model architecture, such as MoE and attention mechanisms, in relation to model quality and training expenses.

• Analyze training jobs and collaborate with systems engineers to improve overall throughput with hardware-aware execution.

• Ensure long training sessions are robust against failures by implementing reliable checkpointing and recovery methods.

• Examine training dynamics and numerical stability, confirming that optimizations maintain convergence.

• Conduct controlled ablation studies to inform larger experiments while ensuring setup and outcomes are reproducible.

• Collaborate with data and evaluation researchers on training mixtures, encompassing midtraining and long-context studies.

• Transition the next open models from architectural studies to full pretraining runs, scaling towards 1–3T parameters.

⛳️ Requirements

• Practical experience in training language models, including modifying training code and analyzing the effects through actual executions.

• Proficient in Python programming.

• Familiarity with PyTorch or a similar framework.

• Adequate systems knowledge to troubleshoot distributed training issues.

• Solid understanding of model architectures and optimization, particularly their impact on convergence.

• Capability to design ablations that address research inquiries and clarify what the results indicate or do not indicate.

• Sound judgment regarding when to conduct investigations independently and when to seek input from a colleague.

🏝️ Benefits

• Equity incentives included in the cash compensation range.

• Flexible work arrangements, allowing the option to work remotely or in-person at our San Francisco offices.

• Visa sponsorship available.

• Relocation support for international candidates.

• Quarterly team off-sites.

• Opportunities for hackathons.

• Access to conferences and learning opportunities.

• Chance to collaborate with a talented, dedicated, and mission-driven team.