Senior Staff Software Engineer, ML Training Infrastructure

Posted 18 hours ago

This is a fully remote position, open to applicants in United States.

📋 Description

• Design and enhance components of a high-performance training platform, encompassing orchestration, scheduling, training abstractions, control plane, observability, and performance tuning.

• Implement comprehensive ML model pipelines that include log processing, feature extraction, dataset schema design/storage, model configuration management, and workflows for model training and profiling/acceleration.

• Monitor and assess the performance of training infrastructure to identify and resolve bottlenecks across all layers of the stack.

• Enhance training throughput across distributed nodes utilizing topology-aware frameworks and hardware solutions such as InfiniBand, NVLink, and NCCL.

• Advocate for system abstractions and tools that empower MLEs to iterate swiftly on models.

• Integrate open-source software tools to allow ML engineers to independently profile and optimize their workflows.

• Uphold a high standard for engineering excellence and foster a culture of quality within the team.

• Lead the design and development of a high-performance multi-tenant AI training platform in collaboration with ML Platform, Infrastructure, Autonomy, and Safety Evaluation teams.


⛳️ Requirements

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related discipline.

• Over 8 years of experience with ML Platforms and the development of ML-based applications (modeling experience is an advantage).

• Demonstrated success in optimizing training through cutting-edge distributed training paradigms.

• Experience in productionizing large-scale data pipelines and data loaders to enhance resource utilization and computational throughput.

• Proficient programming skills in Python, along with prior experience with Lance, PyTorch, Ray, Spark, or similar technologies.

• A proven history of constructing scalable and reliable infrastructure in a dynamic environment while collaborating with MLEs across various modeling teams.

• A thorough understanding of design trade-offs, with the ability to articulate these trade-offs to achieve alignment with cross-functional teams.

• Strong analytical and problem-solving capabilities.

• Excellent verbal and written communication skills, capable of conveying complex technical concepts to non-technical stakeholders.

• Position may necessitate the verification of residence, U.S. person status, and/or citizenship due to national security and export control regulations.


🏝️ Benefits

• Comprehensive health, dental, and vision insurance plans.

• Generous paid time off and flexible working arrangements.

• Opportunities for professional development and continuous learning.

• Collaborative and innovative work environment.

People also viewed

Empower AI13 hours ago

AI/ML Engineer

US flagUnited States OnlyFull-timeMachine Learning Engineer$127.8k – $197.9k/year
ApplyView job
Augeo14 hours ago

Senior Machine Learning Engineer – Applied AI

US flagMinnesota OnlyFull-timeMachine Learning Engineer$150k – $200k/year
ApplyView job
Striveworks18 hours ago

Senior Machine Learning Engineer, Maritime Systems

US flagTexas OnlyFull-timeMachine Learning Engineer$160k – $200k/year
ApplyView job
SandboxAQ1 day ago

Senior Machine Learning Engineer – Medical Devices

US flagUnited States OnlyFull-timeMachine Learning Engineer$131k – $246k/year
ApplyView job
Weekday (YC W21)1 day ago

Machine Learning Engineer, Scientific Computing – AI Evaluation

US flagUnited States OnlyPart-timeMachine Learning Engineer$90 – $140/hour
ApplyView job
Blue Orange Digital1 day ago

Machine Learning Engineer, Mid-level

BR flagBrazil, +6 more countriesFull-timeMachine Learning Engineer$4,000 – $6,000/month
ApplyView job

Never miss a great job!

Get handpicked remote jobs straight to your inbox weekly.

Trusted by 7,400+ designers