Member of Technical Staff – Training Platform

This is a fully remote position, open to applicants in California.

📋 Description

• Develop and manage the hosted training platform for LoRA and comprehensive fine-tuning operations on managed GPU clusters.

• Design and oversee Kubernetes-based training and inference orchestration across multi-cluster, multi-cloud GPU fleets.

• Create and sustain Helm charts for consistent training environments.

• Build Python control-plane agents that monitor pods, report on run status, and synchronize clusters.

• Implement scheduling and autoscaling for diverse H100, H200, and B200 hardware.

• Manage GitOps workflows utilizing pull requests, Helm values, and continuous integration.

• Construct node-local model caches, checkpoint pipelines, and shared storage to facilitate quick cold starts.

• Administer Prometheus, Grafana, Loki, and DCGM for observability infrastructure.

• Develop features for developers including job submission, live run monitoring, logs, metrics, model and adapter management, and comparisons.

• Create FastAPI backend services and REST APIs that connect the platform to operational clusters.

• Develop real-time monitoring and debugging tools, such as streaming logs, step-level metrics, and failure analysis.

• Deliver product UI using Next.js, React, TypeScript, shadcn, Tailwind, tRPC, and TanStack Query.

• Collaborate with RL trainers, inference servers, and environment servers.

• Transform new training capabilities, model architectures, RL algorithms, and operational modes into products.

⛳️ Requirements

• Solid understanding of open model families and fine-tuning techniques (LoRA, QLoRA, full FT, RLHF/RLAIF).

• Familiarity with inference engines (vLLM, SGLang, TensorRT-LLM).

• Knowledge of GPU hardware tradeoffs (H100, H200, B200, NVLink, interconnects, memory hierarchy).

• Comprehension of distributed training principles (data, tensor, pipeline, and expert parallelism; NCCL; multi-node scheduling).

• Extensive experience in Kubernetes operations (Helm, CRDs, operators, KEDA, gang scheduling, GPU operator).

• Proficiency in debugging production clusters using kubectl, with knowledge of pod lifecycle, node troubleshooting, and networking.

• Experience with cloud platforms; GCP preferred, including GCS, GKE, Cloud Run, and Cloud Tasks.

• Background in infrastructure automation with Helm, Terraform, or Ansible and a GitOps mindset.

• Experience in observability tools like Prometheus, Grafana, Loki, OpenTelemetry, and DCGM.

• Fundamental knowledge of Linux including networking, namespaces, and performance tuning.

• Strong Python backend development skills with FastAPI, async programming, and SQLAlchemy.

• Experience in building Python control-plane agents that engage with Kubernetes APIs.

• Modern frontend development experience with TypeScript, React/Next.js, Tailwind, and shadcn.

• Proficiency in REST and tRPC API design.

• Experience in developing developer tools, dashboards, and live-monitoring user interfaces.

• Background in platform or infrastructure development, ideally encompassing both areas.

🏝️ Benefits

• Significant equity.

• Flexible work arrangement (remote or San Francisco office).

• Full visa sponsorship and relocation assistance.

• Professional development budget for courses and conferences.

• Regular team off-sites and attendance at conferences.

• Opportunity to influence the future of decentralized AI development.

• Encouragement of contributions to research and open-source initiatives.