Staff Engineer, Inference Optimizations

This is a fully remote position, open to applicants in Massachusetts.

📋 Description

• Oversee the technical strategy for benchmarking and enhancing performance at the inference engine and GPU kernel levels.

• Enhance attention layers, manage memory and precision, and enable parallel processing across multi-node GPU clusters.

• Boost batch-size performance utilizing AMD’s AITER library for AMD MI355X and optimize CK/ASK kernels for FP8/BF16.

• Discover kernel-fusion potential for GLM-5 Transformer-block kernels, including FlashAttention and RMS Norm.

• Optimize expert gateway router kernels for MoE models such as Qwen3-235B, DeepSeek V3, and GLM-5.

• Act as a subject matter expert on NVIDIA and AMD GPUs along with their software stacks, including CUDA, ROCm, TensorRT, and OpenAI Triton.

• Design and implement quantization techniques, including FP8, INT8, and FP4.

• Lead code and design reviews, mentoring engineers through influence.

• Collaborate with Product Management and TPMs to translate hardware limitations into viable product features.

• Contribute to and integrate open-source AI technologies while participating in GPU infrastructure and model performance communities.

• Direct the technical roadmap for DigitalOcean’s high-performance inference fleet.

⛳️ Requirements

• 5+ years of experience in high-performance computing or AI infrastructure.

• Demonstrated success in resolving compute utilization and memory bandwidth challenges.

• Profound knowledge of the Gen AI landscape, including LLM, VLM, and LMM model families.

• Practical experience with attention-layer optimizations and parallelization in distributed GPU settings.

• In-depth understanding of NVIDIA and AMD GPU architectures and their software ecosystems, including CUDA and ROCm.

• Significant experience in integrating, building, and contributing to open-source software projects.

• Strong system design acumen related to low-level GPU programming, optimization, memory access patterns, and parallel execution.

• Experience serving as a technical lead and facilitating design and delivery through cross-functional collaboration.

• Profound understanding of GPU architectures, including SMs, warp scheduling, and Tensor Cores.

• Expert-level proficiency in Triton or CUDA.

• Advanced GPU kernel development skills; custom CUDA kernels or Triton compiler contributions are preferred.

• Capability to work remotely.

🏝️ Benefits

• Reimbursement for relevant conferences, training, and education, in accordance with company policy.

• Flexible time off.

• Local team meetups.

• Support for mental health and well-being.

• Eligibility for an annual bonus plan for non-sales roles.

• Potential new hire equity grant, subject to plan terms and local eligibility.

• Employee Stock Purchase Program available in participating countries.

• Comprehensive total rewards package.