SGLang HiCache Extends KV Reuse Across Parallelism Configurations
Cache-aware serving is becoming the bottleneck as agentic pipelines resend long prefixes on every call, and HiCache's unified KV layout means cached prefixes survive changes in parallelism setup rather than being discarded on redeployment. Teams running GLM, DeepSeek or other long-context models on SGLang can pair HiCache with external stores like Alibaba Tair KVCache to cut TTFT without re-architecting their serving topology. The iFLYTEK deployment is a concrete reference point for production coding-agent workloads.
LMSYS (SGLang team) published a blog on HiCache, which caches KV data for reuse across context parallelism, pipeline parallelism, speculative decoding and PD Disaggregation serving. The post describes a unified KV layout that enables cache reuse across compatible parallel configurations, targeting agentic workloads that reuse long prefixes across many model calls. In production, iFLYTEK's Astron MaaS team serving GLM-5.1/5.2 with PP, PD Disaggregation and MTP reported cache hit rates rising from 30% to 95% on multi-turn agentic workloads with inputs from 30K to 200K tokens, up to 1M. Median time to first token dropped below 3 seconds from delays that had made the service largely unusable.
HiCache Across Parallelism & Distributed Serving
Agentic workloads reuse long prefixes across many model calls. Our new blog explains how SGLang HiCache makes that reuse work with context parallelism, pipeline parallelism, speculative decoding, and PD Disaggregation serving. It also covers a unified KV layout for reuse across compatible parallel configurations.
For the iFLYTEK Astron MaaS team’s agentic CodingPlan workloads, served by GLM-5.1/5.2 with PP/PD Disaggregation/MTP, the team reports these results after adding HiCache with external storage: · Cache hit rate: 30% → 95% on multi-turn agentic workloads with inputs typically ranging from 30K to 200K tokens, and up to 1M. · Median TTFT: below 3 seconds, down from delays that had made the service largely unusable.
Thanks to @alibaba_cloud Tair KVCache, Alibaba Bailian, @radixark, @NVIDIAAI, @AntGroup and @iflytek Astron MaaS.
Read the full blog: https://t.co/kwIpXluFHc
