Senior Principal Software Engineer

This is a fully remote position, open to applicants in Massachusetts.

📋 Description

• Enhance and implement high-performance LLM inference pipelines.

• Take ownership of inference runtimes across data center, edge, and embedded platforms.

• Improve model performance through techniques such as quantization, kernel fusion, and cache optimization.

• Lead efforts to reduce latency and increase throughput, directly affecting production products.

• Facilitate efficient and dependable deployment without reliance on external vendors.

• Develop deep expertise and take responsibility for: vLLM, TensorRT-LLM, llama.cpp, QAIRT.

• Extend and optimize inference engines using custom CUDA kernels.

• Modify runtimes for constrained and embedded deployment scenarios.

• Implement and assess quantization strategies: INT8, INT4, FP4, FP8, mixed precision AWQ, GPTQ.

• Weigh accuracy, latency, memory usage, and throughput effectively.

• Enhance key-value cache performance through: Paging, Prefix caching, and cache-aware memory layout design.

• Design and refine: Batching strategies, Continuous batching, and Speculative decoding.

⛳️ Requirements

• Demonstrated experience in optimizing ML inference performance in a production setting.

• Strong understanding of GPU architecture and memory hierarchies.

• Practical experience with CUDA and low-level performance optimization.

• Experience in deploying models beyond academic research environments.

• Essential Technical Skills:

• Inference engines: vLLM, TensorRT-LLM, llama.cpp, QAIRT.

• Development and profiling of CUDA kernels.

• Knowledge of quantization techniques: INT8/INT4/FP4/FP8, AWQ, GPTQ.

• Optimization of KV cache and memory layout design.

• Latency optimization skills: batching, speculative decoding, continuous batching.

🏝️ Benefits

• Opportunity for annual bonuses.

• Comprehensive insurance coverage (medical, dental, vision, life, and disability).

• Paid time off.

• Paid holidays.

• Company contributions to the RRSP (Registered Retirement Savings Plan).

• Equity awards available for certain roles and levels.

• Options for remote and/or hybrid work depending on the position.