Machine Learning Engineer – ML Training Platform

This is a fully remote position, open to applicants in United States, +1 more country.

📋 Description

• Design, develop, and enhance the platform that facilitates ongoing experimentation and extensive training on consumer nodes and cloud instances.

• Create resource management systems for provisioning and orchestrating computing across AWS, GCP, and Azure utilizing Pulumi/Terraform.

• Manage dynamic scaling, state synchronization, and concurrent operations across numerous diverse nodes.

• Engineer a fault-tolerant distributed machine learning infrastructure for GPU clusters using NVIDIA runtime.

• Implement S3 checkpointing, manage large datasets and streaming, monitor health, and develop resilient retry strategies.

• Construct systems that simulate and manage bandwidth shaping, latency injection, and packet loss.

• Oversee node churn and ensure continuous data flow across workers with varying connectivity.

⛳️ Requirements

• Proven production experience with infrastructure-as-code tools (Pulumi/Terraform/CloudFormation) in managing multi-cloud deployments.

• Proficiency with Docker/Kubernetes (EKS), GPU workloads, and large-scale heterogeneous clusters.

• Knowledge of distributed training workflows, including checkpointing, data sharding, model versioning, and orchestration of long-running jobs.

• Familiarity with decentralized networking concepts, such as P2P, NAT traversal, traffic shaping, and real bandwidth limitations.

• Strong skills in Python engineering, particularly with asyncio, concurrency, retry logic, cloud SDKs, and CLI tooling.

• Practical experience in observability and SRE practices, including Prometheus/Grafana, performance profiling, and incident management.

• Background in a startup environment with significant service orchestration experience or at a large tech company.

• Capability to showcase the systems you have owned.

• A belief in Protocol Learning as a promising approach for collective, trustless, and sovereign AI.

• Professional-level proficiency in English, both written and spoken.

• Willingness to collaborate across various time zones.

🏝️ Benefits

• Substantial equity/ownership opportunities for key technical contributors, in addition to a competitive base salary.

• Flexible work environment with a globally distributed team.

• Option for full visa sponsorship and relocation assistance to either Australia or the US.

• Chance to engage with innovative, unpublished challenges in the training and serving of frontier models.