Senior ML Engineer – Inference Stack Optimization

This is a fully remote position, open to applicants in Germany, +7 more countries.

📋 Description

• Enhance the efficiency of the inference-stack regarding throughput, latency, cold starts, and cost per token.

• Adjust continuous batching, chunked prefill, speculative decoding, CUDA-graph capture, and scheduling across GPU fleets.

• Improve the time to first token and inter-token latency by analyzing compute, memory bandwidth, scheduling, storage, and network bottlenecks.

• Boost KV-cache utilization through hierarchical and paged attention, prefix caching, eviction policies, and quantized KV.

• Mitigate cold starts utilizing O_DIRECT, fastsafetensors-class weight loaders, JIT and compile caches, and shared-filesystem read paths.

• Quantize weights, activations, and KV using INT8/INT4 and FP8/FP4 while maintaining quality in real workloads.

• Scale distributed inference across multiple nodes through network-aware placement, prefill/decode disaggregation, and non-bottlenecking checkpoint strategies.

• Set the technical trajectory by selecting benchmarks, technologies to implement, and systems to develop; back decisions with documentation and reproducible experiments.

⛳️ Requirements

• Over 5 years of experience in developing real ML systems.

• A portfolio that showcases expertise in inference or training infrastructure beyond model training notebooks.

• Proficient in production-grade Python for service development.

• Hands-on experience with vLLM, SGLang, or TensorRT-LLM.

• Solid understanding of engine performance across various GPUs.

• Experience in measuring quantization regressions.

• Familiarity with multi-GPU and multi-node configurations, including tensor and expert parallelism and collective communication.

• Knowledge of page caches, O_DIRECT, network filesystems, and handling terabyte-scale reads.

• Capability to instrument systems prior to optimization and differentiate genuine enhancements from benchmark artifacts.

• Experience with LLM routing, RouterBench-class evaluations, cost/quality/latency Pareto fronts, calibration, abstention, conformal methods, online A/B loops, and drift monitoring is a plus.

• Completion of the hiring process is required, including recruiter screening, hiring manager interview, technical system design interview, live coding, and executive culture check.

• A background check may be performed during the final recruitment phase.

• Please note that Cast AI does not offer visa sponsorship or work permits.

🏝️ Benefits

• Competitive salary based on experience level.

• Flexible, remote-first global work environment.

• Equity options available.

• 10% of work time allocated for personal projects or self-improvement.

• Learning budget for professional and personal growth, including international conferences and courses.

• Annual hackathon event.

• Team-building budget and company events.

• Equipment budget for necessary tools.

• Additional vacation days off.