
Senior Staff Software Engineer, ML Training Infrastructure
Posted 18 hours ago

Posted 18 hours ago
This is a fully remote position, open to applicants in United States.
• Design and enhance components of a high-performance training platform, encompassing orchestration, scheduling, training abstractions, control plane, observability, and performance tuning.
• Implement comprehensive ML model pipelines that include log processing, feature extraction, dataset schema design/storage, model configuration management, and workflows for model training and profiling/acceleration.
• Monitor and assess the performance of training infrastructure to identify and resolve bottlenecks across all layers of the stack.
• Enhance training throughput across distributed nodes utilizing topology-aware frameworks and hardware solutions such as InfiniBand, NVLink, and NCCL.
• Advocate for system abstractions and tools that empower MLEs to iterate swiftly on models.
• Integrate open-source software tools to allow ML engineers to independently profile and optimize their workflows.
• Uphold a high standard for engineering excellence and foster a culture of quality within the team.
• Lead the design and development of a high-performance multi-tenant AI training platform in collaboration with ML Platform, Infrastructure, Autonomy, and Safety Evaluation teams.
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related discipline.
• Over 8 years of experience with ML Platforms and the development of ML-based applications (modeling experience is an advantage).
• Demonstrated success in optimizing training through cutting-edge distributed training paradigms.
• Experience in productionizing large-scale data pipelines and data loaders to enhance resource utilization and computational throughput.
• Proficient programming skills in Python, along with prior experience with Lance, PyTorch, Ray, Spark, or similar technologies.
• A proven history of constructing scalable and reliable infrastructure in a dynamic environment while collaborating with MLEs across various modeling teams.
• A thorough understanding of design trade-offs, with the ability to articulate these trade-offs to achieve alignment with cross-functional teams.
• Strong analytical and problem-solving capabilities.
• Excellent verbal and written communication skills, capable of conveying complex technical concepts to non-technical stakeholders.
• Position may necessitate the verification of residence, U.S. person status, and/or citizenship due to national security and export control regulations.
• Comprehensive health, dental, and vision insurance plans.
• Generous paid time off and flexible working arrangements.
• Opportunities for professional development and continuous learning.
• Collaborative and innovative work environment.
Empower AI
Augeo
Striveworks
SandboxAQ
Get handpicked remote jobs straight to your inbox weekly.