This reframes context engineering as a systems problem: instead of editing the visible prompt, the agent rewrites its own cache, avoiding the re-prefill penalty that summarization triggers. Teams building long-running agents with multi-turn tool calls can cut serving cost measurably without changing what the model sees. The gains apply to any stack serving LLMs with prefix caching, such as vLLM-based deployments.
一个 3B 的线性注意力模型，每条序列状态固定 29.5 MB，不随上下文膨胀，还全程跑在 Trainium2 上，长上下文推理的内存账很好算。
vLLM now spans three hardware ecosystems, which reduces the lock-in that previously tied inference stacks to NVIDIA GPUs. Teams running self-hosted inference can benchmark vLLM against vendor-specific runtimes on AMD and TPU infrastructure. The KV cache management work matters most for serving long-context models where memory, not compute, is the bottleneck.
Shifting KV-cache transfer coordination off the critical path removes a hidden latency tax in prefill-decode disaggregation, and the 3x TTFT gain applies to any team serving LLMs at scale on vLLM. Practitioners running inference on AWS can adopt the NIXL connector on EC2 P5en with EFA networking directly from the guide. Compared with pull-based transfer, the push design needs no receiver-side polling, which is what unlocks the latency savings.
Percy Liang's closing argument reframes the KV cache as the structural root of Transformer inference inefficiency, pointing to architectures designed for inference from day one, such as SSMs, linear attention, and diffusion-based non-autoregressive generation. Engineers building or tuning inference stacks get a single coherent map from architecture to serving systems. The executable lecture notes with sympy-verified formulas make the material directly reproducible rather than slide-level theory.
清华 THUNLP 开源了个 KV cache 压缩方法，压到 1/32 还能在 RULER 上保持 96.7% 准确率，比 KVZip+ 的 40% 高出一大截，长文档检索场景可以直接试。
The finding suggests viral multi-agent memory demos may not transfer to standard evaluation setups, since benchmarks are structured in ways that neutralise recall benefits. Engineers deciding whether to add persistent memory to agent pipelines should weigh the near-zero measured accuracy gain against its efficiency advantages. The paper's benchmark-design critique is a useful checklist when validating similar architectures.
KV-cache reuse, not model quality, may become the bottleneck for agent products at scale, since idle context still occupies memory while agents wait on tool calls. Engineers building multi-step agent backends can use Dynamo's router-plus-cache-manager design as a reference for cutting serving cost without adding GPUs. It competes in the same layer as vLLM and SGLang-style serving stacks.
SWA bounded replay shows how KV-cache reuse for agent loops can be engineered around sliding-window attention, a pattern many teams will copy as agent traffic grows. Engineers running agent stacks on vLLM can apply it to reduce TTFT under high concurrency. The writeup also signals that DeepSeek-V4.1-Flash inference performance is being actively tuned by the community rather than relying on the lab alone.
The three reuse tricks all called 'looped transformers' differ in cache behaviour and cost, and picking the wrong one can burn a training week or budget a 3B decode like a 3B file. Builders planning efficient architectures can use this to decide whether extra loops justify their compute and whether a second cache is needed. Nanbeige 4.2 offers a concrete open reference point against standard Transformers.
The key insight is that near-fixed-point recurrent states make much of the compute spent on full backpropagation and full KV caches redundant, cutting costs across training, decoding and RL at once. Practitioners building efficient inference stacks or RL pipelines for recurrent models can apply KV sharing and distilled prefill directly. Compared with Huginn's fixed prior, the learned prior plus orthogonal injection improves perplexity at every scale tested up to 1.6B parameters.
一篇新论文 CompKV，重新设计了稀疏注意力里的 token 选择逻辑，长上下文推理自注意力最高快 6.85 倍，做推理优化的可以看看。
DeepSeek 新出的 V4.1 Flash 模型，性能不错，和 GLM-5.3 差不多，关键是内存消耗特别低，适合做 agentic 任务，价格也便宜很多。
