Staff Engineer, Inference Optimizations

This is a fully remote position, open to applicants in Colorado.

📋 Description

• Direct the technical strategy for benchmarking and enhancing performance at the inference engine and GPU kernel levels.

• Develop solutions for optimizing attention layers, managing memory and precision, and implementing advanced parallelization across multiple GPU clusters.

• Apply state-of-the-art optimization methods for generative AI inference.

• Enhance batch-size performance utilizing AMD’s AITER library for AMD MI355X and fine-tune CK or ASK for FP8/BF16.

• Discover opportunities for kernel fusion within GLM-5 Transformer-block layers, including FlashAttention and RMS Norm.

• Optimize expert gateway router kernels for MoE models like Qwen3-235B, DeepSeek V3, and GLM-5.

• Provide guidance on the procurement of modern NVIDIA and AMD GPU hardware as well as software integration.

• Create and implement quantization techniques such as FP8, INT8, and experimental FP4.

• Lead code and design reviews without direct management responsibilities.

• Collaborate with Product Management and TPMs to transform hardware constraints into deliverable product features.

• Contribute to and integrate open-source AI initiatives and engage in GPU infrastructure and model-performance communities.

⛳️ Requirements

• Over 5 years of experience in high-performance computing or AI infrastructure.

• Demonstrated success in resolving compute utilization and memory bandwidth issues.

• In-depth knowledge of the Generative AI landscape, including LLM, VLM, and LMM model families.

• Practical experience with attention-layer optimizations and parallelization in distributed GPU settings.

• Thorough understanding of NVIDIA and AMD GPU architectures and their software ecosystems, including CUDA and ROCm.

• Significant experience in integrating, developing, and contributing to open-source software projects.

• Strong system design capabilities related to low-level GPU programming, optimization, memory access patterns, and parallel execution.

• Experience serving as a technical lead and facilitating design and delivery through cross-functional collaboration.

• Profound understanding of GPU architectures, including SMs, warp scheduling, and Tensor Cores.

• Expert-level proficiency in Triton or CUDA.

• Experience contributing to the Triton compiler or creating custom CUDA kernels for a major LLM is preferred.

🏝️ Benefits

• Reimbursement for relevant conferences, training, and education, in accordance with company policy.

• Flexible vacation policy.

• Opportunities for local team meetups.

• Support for mental health and well-being.

• Eligibility for DigitalOcean’s annual bonus plan.

• Potential eligibility for a new hire equity grant.

• In applicable countries, the chance to participate in the Employee Stock Purchase Program.

• Commitment to equal-opportunity employment.