Staff/Principal Software Engineer – AI Platform Model Serving

This is a fully remote position, open to applicants in United States.

📋 Description

• Redesign Netflix’s AI/ML model serving platform and infrastructure to support real-time, asynchronous, and agentic workloads.

• Integrate in-house inference infrastructure, external model hosting, and third-party LLM providers through a serving, routing, and experimentation layer.

• Facilitate consistent deployment, routing, and A/B testing of models, irrespective of their hosting locations.

• Reassess GPU capacity provisioning and utilization, including reallocating off-peak production capacity for internal coding and developer productivity initiatives.

• Spearhead the technical architecture and strategic direction for Netflix’s GenAI and Foundation Model serving initiatives.

• Tackle challenges related to disaggregated serving, intelligent routing, speculative decoding, prompt caching, and multi-GPU inference.

• Incorporate AI capabilities into member-facing products at a global scale.

• Develop robust, resumable execution for long-duration agentic workloads.

• Establish and take ownership of the technical direction for a growing engineering team.

• Guide architectural choices and ensure alignment across various teams.

• Formulate long-term technical roadmaps while ensuring scalability, maintainability, and alignment with business goals.

• Collaborate with product managers, operations, and executive stakeholders.

• Create architecture decision records and technical proposals.

⛳️ Requirements

• BS/MS/PhD in Computer Science, Applied Mathematics, Engineering, or a related discipline.

• Over 12 years of pertinent experience in AI/ML Infrastructure and distributed systems.

• Proven experience with high-traffic distributed services and AI/ML model inference infrastructure.

• Experience in developing scalable Foundation Model serving solutions.

• Practical expertise in minimizing inference latency and costs.

• Knowledge of NVIDIA Triton Inference Server, vLLM, or SGLang.

• Experience in scaling AI/ML models using Kubernetes.

• Familiarity with AWS, Azure, or GCP.

• In-depth knowledge of distributed systems and high-performance RPC.

• Expertise in GPU/CPU inference optimization, control planes, and experimentation infrastructure.

• Proficiency with contemporary LLM serving stacks.

• Proven track record in architecting systems at an extreme scale.

• Experience ensuring high availability, security, and performance.

• Ability to influence architectural decisions across different teams.

• Capability to shape long-term technical roadmaps.

• Experience collaborating with product managers, operations, and executive stakeholders.

• Strong communication skills to convey complex technical concepts and produce ADRs and proposals.

• Contributions to open-source projects in AI/ML, distributed systems, or infrastructure (preferred).

• Hands-on experience with AI model building and optimization techniques such as quantization, model compression, or model pruning (preferred).

• Proficiency with AI coding tools (preferred).

• Experience with model gateways and intelligent routing (preferred).

• Familiarity with multi-adapter model serving, disaggregated prefill/decode, or large-scale KV cache management (preferred).

• Experience using evaluation and benchmarking tools for LLMs and AI agents (preferred).

• Knowledge of GPU topology, NVIDIA CUDA programming, and machine learning compilers (preferred).

• Familiarity with PyTorch, TensorFlow, JAX, Hugging Face Transformers, vLLM, SGLang, TensorRT-LLM, TGI, DeepSpeed-Inference, NVIDIA Dynamo, CUDA, NCCL, and NVIDIA Nsight (preferred).

🏝️ Benefits

• Health Plans

• Mental Health support

• 401(k) Retirement Plan with employer match

• Stock Option Program

• Disability Programs

• Health Savings and Flexible Spending Accounts

• Family-forming benefits

• Life and Serious Injury Benefits

• Paid leave of absence programs

• Full-time hourly employees earn 35 days annually for paid time off to be utilized for vacation, holidays, and sick leave.

• Full-time salaried employees are granted immediate access to flexible time off.