Staff Software Engineer, AI Inference

This is a fully remote position, open to applicants in United States.

📋 Description

• Oversee the design and development of our production inference platform.

• Establish the technical roadmap for inference infrastructure, model serving, and runtime optimization.

• Create and manage scalable, cost-efficient systems for deploying large language models in a production environment.

• Assess and incorporate contemporary inference technologies, frameworks, and serving runtimes.

• Enhance latency, throughput, GPU utilization, memory efficiency, and overall infrastructure costs.

• Construct systems for model deployment, traffic management, autoscaling, scheduling, observability, and operational excellence.

• Collaborate with ML engineers to transition new models and inference techniques into production.

• Develop benchmarking methodologies to assess new models, runtimes, and hardware.

• Make critical architectural decisions regarding when to develop internally versus utilizing open-source or commercial solutions.

• Guide engineers as the team expands and assist in establishing engineering best practices for AI infrastructure.

⛳️ Requirements

• Extensive experience in designing and managing production AI inference systems.

• Background in building or leading production LLM serving infrastructure.

• In-depth experience with one or more modern inference runtimes and frameworks such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, Hugging Face TGI, NVIDIA Dynamo, or similar technologies.

• Strong expertise in distributed systems, backend infrastructure, or high-performance platform engineering.

• Experience in optimizing inference performance across GPU workloads, focusing on latency, throughput, batching, memory utilization, and serving efficiency.

• Proficient in operating GPU infrastructure in a production setting.

• High proficiency in Python and at least one systems programming language (such as Go, Rust, or C++).

• Demonstrated ability to lead technical architecture for complex infrastructure projects.

• Exceptional communication skills with the capability to influence technical direction across engineering teams.

🏝️ Benefits

• Health insurance