Staff Engineer, Inference Optimizations

This is a fully remote position, open to applicants in Texas.

📋 Description

• Spearhead the technical strategy for benchmarking and enhancing performance at the inference engine and GPU kernel layers.

• Design solutions for optimizing attention layers, managing memory and precision, and implementing advanced parallelization across multi-node GPU clusters.

• Apply state-of-the-art optimization techniques tailored for generative AI workloads.

• Enhance batch size performance utilizing AMD’s AITER library for AMD MI355X, including the fine-tuning of CK or ASK for FP8/BF16.

• Discover kernel fusion opportunities for GLM-5 Transformer-block kernels, such as FlashAttention and RMS Norm.

• Optimize expert gateway router kernels for MoE models including Qwen3-235B, DeepSeek V3, and GLM-5.

• Provide guidance on contemporary NVIDIA and AMD GPU families, associated software stacks, hardware procurement, and software integration.

• Create and implement quantization methods including FP8, INT8, and experimental FP4.

• Lead code and design reviews while offering technical mentorship without direct management responsibilities.

• Collaborate with Product Management and Technical Program Managers (TPMs) to convert hardware limitations into actionable product features.

• Contribute to and integrate open-source AI initiatives while maintaining an active presence in GPU infrastructure and model-performance communities.

⛳️ Requirements

• Over 5 years of experience in high-performance computing or AI infrastructure.

• Demonstrated ability to resolve compute utilization and memory bandwidth constraints.

• Thorough knowledge of the Generative AI landscape, encompassing LLM, VLM, and LMM model families.

• Practical experience with attention-layer optimizations and parallelization techniques within distributed GPU environments.

• In-depth understanding of NVIDIA and AMD GPU architectures and software ecosystems, including CUDA and ROCm.

• Significant experience in integrating, building, and contributing to open-source software projects.

• Strong system design capabilities related to low-level GPU programming, optimization, memory access patterns, and parallel execution.

• Proven experience in a technical lead role, driving design and delivery through cross-functional collaboration.

• Profound comprehension of GPU architectures, including SMs, Warp scheduling, and Tensor Cores.

• Expert-level proficiency in Triton or CUDA.

• Experience in contributing to the Triton compiler or developing custom CUDA kernels for a major LLM is preferred.

🏝️ Benefits

• Reimbursement for relevant conferences, training, and educational opportunities, in accordance with company policy.

• Flexible time off.

• Local team gatherings.

• Support for mental health and well-being.

• Eligible employees may receive equity compensation, including a grant for new hires.

• Eligible employees may take part in the Employee Stock Purchase Program in qualifying countries.

• Participation in DigitalOcean’s annual bonus plan.

• Commitment to equal-opportunity employment.