GLM-5.3 长上下文服务优化：远程 KV 缓存分片与异步加载实践
When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
给做大模型推理服务的同学：GLM-5.3 上远程 KV 缓存反而把 p99 拖到 83 秒的踩坑实录，还给了带宽阈值公式和修复方案。
研究团队在 8xB300 节点上用 PCP8 部署 FP8 MoE 模型 GLM-5.3，测试 LMCache + Valkey 远程 KV 缓存层，发现同步加载使 16k token 的 p99 从 3.96 秒涨到 17.3 秒。实测一次远程命中耗时 0.88 秒，而重新计算 prompt 只需 0.31 秒，p99 飙到 83.32 秒时 GPU 利用率仅 7%。作者推导出成本模型：命中仅在带宽超过 2.53 GB/s 时划算，且由于每次命中约 0.33 秒的固定开销，短于约 17k token 的 prompt 无论如何都不该走远程缓存。按 rank 分片后 GET 带宽提升 6.3 倍至 10.76 GB/s，配合异步加载，命中率 98.5% 的流的 p99 从 83.32 秒降到 0.67 秒。分片方案让 prefill 吞吐提升 14%，叠加上游 kernel 修复后达 34%，相关改动已提交给 LMCache 上游。
When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
Reusing the attention KV cache across requests is a standard way to cut prefill cost in long-context LLM serving, and production systems increasingly back the per-engine cache with a remote tier shared by every engine. We study such a tier in a disaggregated deployment of GLM-5.3, an FP8 mixture-of-experts model with multi-head latent attention (MLA), on 8xB300 nodes running prefill context parallelism (PCP8). Enabling the remote tier (LMCache with a Valkey store) left median time-to-first-token unchanged but raised the 16k-token p99 from 3.96 s to 17.3 s. Replaying a production request stream on an isolated engine, we find that a remote hit took 0.88 s against 0.31 s to recompute the prompt, and that synchronous loading turned a hit-heavy stream into an unstable queue: p99 83.32 s with the GPUs 7% busy. A two-term cost model explains both: a hit pays off only if the tier delivers more than B* = sR (2.53 GB/s for this model and GPU), and a fixed per-hit engine cost (T0 ~ 0.33 s) means that hits on prompts shorter than about 17k tokens cannot beat recompute at any bandwidth. Guided by the model, we (i) shard the MLA KV store across PCP ranks, with a proof that the existing minimum-over-ranks lookup returns the longest cached prefix and a load protocol that can never desynchronize collectives; (ii) partition the remote tier by rank, one server instance per rank with no coordination, which raises GET bandwidth 6.3x to 10.76 GB/s; (iii) add asynchronous loading to the sharded store; and (iv) fix three client bugs that caused unbounded stalls. On the hit-heavy stream, with 98.5% of prompt tokens served from the remote tier, p99 falls from 83.32 s to 0.67 s. The sharded store raises prefill throughput by 14%, and by 34% together with an upstream kernel fix. All changes are submitted upstream to LMCache.
