#VelServe

1 stories · 1 in 7 days · 1 in 30 days
Stories, products and papers tagged "VelServe", newest first.
Nov 16
PapersBuildersSelected13:20
VelServe scheduler cuts tail latency in LLM decode serving

The non-obvious insight is that tail calibration, not average throughput, is the right knob for meeting streaming SLOs in production inference. Engineers building interactive LLM services or operating disaggregated prefill/decode stacks can apply its scheduling approach directly to reduce SLO violations without extra capacity. The evaluation covers open-weight models widely deployed in the region, including Qwen3-8B and DeepSeek variants, making the results directly comparable for teams benchmarking serving layers against vLLM-style baselines.

Event
arXiv: DeepSeek@Ishaan Datta et al. (5)Source