Senior / Lead Machine Learning Engineer, Inference

This is a fully remote position, open to applicants in Germany.

📋 Description

• Develop and enhance real-time inference systems for consumer-oriented AI applications.

• Collaborate with first-party speech models, LLM serving, and modular high-volume real-time APIs.

• Improve model serving and inference efficiency using contemporary serving frameworks and acceleration methods.

• Analyze and optimize code and NVIDIA GPU workloads to enhance latency and throughput.

• Scale inference across Kubernetes, Ray, multi-GPU, and multi-node environments.

• Manage thousands of simultaneous connections with reliability.

• Transition models from research through containerization, serving optimization, and deployment in production.

• Engage daily with leadership and engineering teams based in the US.

• Create benchmarks or prototypes to address ambiguous performance and reliability issues.

• Contribute to significant engineering efforts and, when appropriate, to open-source projects and technical documentation.

⛳️ Requirements

• In-depth knowledge of contemporary serving frameworks and techniques such as vLLM or TRT-LLM.

• Practical experience with quantization, distillation, caching strategies, continuous batching, paged attention, and speculative decoding.

• Expertise in C++, CUDA, Rust, or highly optimized Python.

• Capability to profile code and enhance NVIDIA GPU performance.

• Familiarity with Kubernetes, Ray, custom load balancing, multi-GPU/multi-node inference, and managing thousands of concurrent connections.

• Experience with substantial systems programming projects, open-source contributions to major inference engines, or in-depth technical write-ups.

• Proficient in taking a model from the research team through containerization, serving optimization, and reliable deployment in production.

• PhD in Computer Science, Physics, Mathematics, or equivalent practical experience in building backend or machine learning systems.

• Professional proficiency in English, both written and spoken, is mandatory.

• Eligibility to work legally in Germany, as indicated by the required legal-right-to-work application question.

🏝️ Benefits

• Full U.S. visa and relocation assistance may be available for candidates looking to move to the San Francisco Bay Area in the future, contingent upon business needs and applicable legal and work authorization requirements.