Staff Engineer, Inference Optimizations

This is a fully remote position, open to applicants in California.

📋 Description

• Direct the technical strategy focused on benchmarking and enhancing performance at the inference-engine and GPU-kernel levels.

• Address intricate performance challenges related to attention layers, memory management, precision control, and parallel processing across multi-node GPU clusters.

• Apply optimization methods for generative AI inference tasks.

• Enhance batch-size performance utilizing AMD's AITER library, and optimize CK or ASK kernels for FP8/BF16.

• Recognize kernel-fusion possibilities within Transformer-block layers, including FlashAttention and RMS Norm.

• Optimize expert-gateway router kernels for Mixture of Experts (MoE) models such as Qwen3-235B, DeepSeek V3, and GLM-5.

• Provide guidance on the procurement of NVIDIA/AMD hardware and software integration.

• Design and implement quantization methods, including FP8, INT8, and FP4.

• Lead code and design evaluations while raising the technical standards of the team.

• Collaborate with Product Management and Technical Program Managers (TPMs) to transform hardware limitations into marketable product features.

• Engage in and contribute to open-source AI initiatives while maintaining an active role in GPU infrastructure and model-performance communities.

⛳️ Requirements

• Over 5 years of experience in high-performance computing or AI infrastructure.

• Demonstrated success in resolving compute utilization and memory bandwidth challenges.

• Profound understanding of the Generative AI landscape, including LLM, VLM, and LMM model families.

• Practical experience with attention-layer optimizations and parallel processing in distributed GPU setups.

• In-depth knowledge of NVIDIA and AMD GPU architectures and software ecosystems, including CUDA and ROCm.

• Significant experience in integrating, building with, and contributing to open-source software initiatives.

• Exceptional skills in system design concerning low-level GPU programming, optimization, memory access patterns, and parallel execution.

• Experience in a technical leadership role, facilitating design and delivery through cross-functional collaboration.

• Comprehensive understanding of GPU architectures, encompassing SMs, warp scheduling, and Tensor Cores.

• Advanced proficiency in Triton or CUDA.

• Experience with contributions to the Triton compiler or development of custom CUDA kernels for a major LLM is preferred.

🏝️ Benefits

• Reimbursement for relevant conferences, training, and educational pursuits, in accordance with company policy.

• Flexible vacation policy.

• Opportunities for local team gatherings.

• Support for mental health and well-being.

• Eligibility for an annual bonus plan for non-sales positions.

• Eligible employees may receive equity compensation, including a new hire equity grant.

• Employee Stock Purchase Program available in participating countries.

• Access to professional development resources.

• Commitment to equal-opportunity employment.