ProductsSelected11月5日 00:38

Alibaba Tair team co-leads SGLang agentic cache scheduling roadmap

官方账号阿里云 Alibaba Cloud@alibaba_cloud

Post

Why we picked it

KV cache reuse is becoming the bottleneck for agent serving, and engine-level cache sharing across parallelism setups is a practical lever for cutting latency and GPU cost. Teams running SGLang inference at scale can apply HiCache to avoid recomputing prefixes in multi-parallel, distributed deployments. Alibaba shaping an open-source inference stack also signals how Chinese cloud vendors compete at the serving layer.

Alibaba Cloud's ApsaraDB Tair KVCache team is co-leading SGLang's Agentic Cache Scheduling roadmap together with the SGLang community. The work integrates HiCache with parallel computation and distributed inference, and develops cache sharing and scheduling across serving engines to reduce repeated work and response times for agentic workloads. A new SGLang blog explains how HiCache works with context parallelism, pipeline parallelism, speculative decoding, and prefill-decode disaggregation. It also introduces a unified KV layout for sharing stored cache data across compatible parallel configurations.

Original · 阿里云 Alibaba Cloud

ApsaraDB's Tair KVCache team is co-leading SGLang's Agentic Cache Scheduling roadmap with the SGLang community. Together, we’re integrating HiCache with parallel computation and distributed inference, and developing cache sharing and scheduling across serving engines to reduce repeated work and response times for agentic workload.

The latest SGLang blog explains how HiCache works with context parallelism, pipeline parallelism, speculative decoding, and prefill-decode disaggregation. It also introduces a unified KV layout for sharing stored cache data across compatible parallel configurations.

Read the full blog 👇 HiCache Across Multi-Parallelism and Distributed Serving: https://t.co/xO5SLcz7w6 #AlibabaCloud #ApsaraDB #Tair #SGLang #Radixark

#SGLang #Alibaba Cloud #Tair #HiCache #builders

Keep reading

Recent stories with the same tag or section

Papers Builders13:10

Chinese contentLoopVQ：面向 Looped Transformer 的亚 1-bit KV Cache 量化方法

.looped 模型参数省但 KV cache 爆炸，这篇提出 0.5bit 量化方案，SGLang 实测吞吐最高涨 6.1 倍，做部署的可以看看。

#LoopVQ #KV Cache #量化 #Looped Transformer

arXiv cs.LG@Minsoo Cheong, Sungjoo YooSource

Papers BuildersSelected11:52

LeanSparse Cuts GPU Memory for Sparse-Attention LLM Serving

The key insight is that prefill memory spikes, not just KV cache size, throttle concurrency — streaming layer histories through reusable buffers removes that admission bottleneck. Engineers running long-context inference on SGLang can adopt the design to serve more users per GPU without changing model weights. Compared with HiSparse and ECHO, the gains come from better indexing rather than more hardware.

#LeanSparse #DeepSeek #SGLang #sparse attention

arXiv: DeepSeek@Zaifeng Pan et al. (6)Source

Papers US vs ChinaSelected10:50

Multi-layer Indexer Speeds DeepSeek Sparse Attention Decode

The indexer scan, not sparse attention itself, turns out to be the long-context bottleneck, and reusing selections across layers removes it without any retraining. Teams serving DeepSeek-V3.2 or GLM-5 long-context workloads on SGLang can adopt MLI for faster decode at unchanged accuracy, and it composes with the IndexShare reuse GLM-5.2 already ships.

#DeepSeek #DeepSeek Sparse Attention #GLM-5 #SGLang

arXiv: DeepSeek@Serdarcan Dilbaz et al. (4)Source

How-to BuildersSelected04:07

SGLang HiCache Extends KV Reuse Across Parallelism Configurations

Cache-aware serving is becoming the bottleneck as agentic pipelines resend long prefixes on every call, and HiCache's unified KV layout means cached prefixes survive changes in parallelism setup rather than being discarded on redeployment. Teams running GLM, DeepSeek or other long-context models on SGLang can pair HiCache with external stores like Alibaba Tair KVCache to cut TTFT without re-architecting their serving topology. The iFLYTEK deployment is a concrete reference point for production coding-agent workloads.

#SGLang #HiCache #GLM-5.1 #iFLYTEK