Senior Machine Learning Engineer, Inference

This is a fully remote position, open to applicants in Serbia.

📋 Description

• Enhance inference processes by utilizing contemporary serving frameworks and methodologies.

• Speed up models through techniques such as quantization, distillation, caching, continuous batching, paged attention, and speculative decoding.

• Analyze and optimize high-performance code specifically for NVIDIA GPUs.

• Develop and expand distributed inference systems leveraging Kubernetes, Ray, custom load balancing, and multi-GPU/multi-node setups.

• Manage thousands of simultaneous connections with reliability.

• Transition models from the research team through containerization, optimization for serving, and deployment in production.

• Create benchmarks or prototypes to address ambiguous technical challenges.

• Evaluate architectures to enhance latency and throughput.

• Ensure that engineering outputs are deployed and remain stable.

• Engage in daily collaboration with leadership and engineering teams based in the U.S.

• Contribute to public technical documentation and open-source projects as appropriate.

⛳️ Requirements

• Profound knowledge of contemporary serving frameworks and techniques, including vLLM or TRT-LLM.

• Practical experience with quantization, distillation, caching techniques, continuous batching, paged attention, and speculative decoding.

• Expertise in C++, CUDA, Rust, or highly optimized Python programming.

• Capability to profile code and enhance performance on NVIDIA GPUs.

• Experience working with Kubernetes, Ray, custom load balancing, multi-GPU/multi-node inference, and managing thousands of concurrent connections.

• Involvement in significant systems programming projects, contributions to major open-source inference engines, or detailed technical write-ups.

• Ability to transition a model from the research phase through containerization and optimization for reliable production deployment.

• A PhD in Computer Science, Physics, or Mathematics, or equivalent hands-on experience in building backend or ML systems.

• Professional proficiency in English, both written and spoken.

• Quick learning ability, capability to thrive in uncertain environments, work autonomously, and prioritize impactful and stable production delivery.

🏝️ Benefits

• Comprehensive U.S. visa and relocation assistance may be offered to candidates considering a move to the San Francisco Bay Area in the future, contingent on business requirements and relevant legal and work authorization stipulations.

• Opportunities to present work and contribute to open-source initiatives.