AITOPAI热报
PapersSelected

AKV: Agents Manage Their Own KV Cache to Cut LLM Costs

DAIR.AI@dair_ai
Post
View the post
Why we picked it

This reframes context engineering as a systems problem: instead of editing the visible prompt, the agent rewrites its own cache, avoiding the re-prefill penalty that summarization triggers. Teams building long-running agents with multi-turn tool calls can cut serving cost measurably without changing what the model sees. The gains apply to any stack serving LLMs with prefix caching, such as vLLM-based deployments.

A new paper introduces cache engineering, a technique that lets long-running agents operate directly on their own KV cache. The agent can remove entries from future attention and shift the positions of remaining keys, while leaving the trajectory text unchanged, providing a programmable interface for refining cache strategies. Applying recursive self-improvement to cache policies reduces cost by 7.1% over expert design. A rolling drop of old tool responses scores 1.7 points higher and costs 27.7% less than threshold-triggered summarization. The accompanying AKV serving engine delivers up to 1.2x throughput and 1.5x faster P95 latency.

Original · DAIR.AI

We love papers that go deeper down the stack.

This is a good example that shifts from agentic context engineering to agentic cache engineering.

They also apply recursive self-improvement to cache policies to further reduce cost by 7.1% over expert design.

Specifically, this work focuses on KV cache management in long-running agents.

When an agent summarizes or trims its history, the prefix change invalidates the cache after it, and the retained context must be prefilled again.

Cache engineering gives the agent two operations on its own KV cache. One removes entries from future attention, and the other shifts the positions of the keys that remain. The trajectory text stays unchanged. This essentially provides a programmable interface for agents to define and refine their own cache engineering strategies.

A rolling drop of old tool responses scores 1.7 points higher and costs 27.7% less than threshold-triggered summarization. Their AKV serving engine adds up to 1.2x throughput and 1.5x faster P95 latency.

https://t.co/fks8lJ46P3

Chat with Paper: https://t.co/wnCyl6Oxqi

#KV cache#agents#context engineering#reasoning models

Keep reading

Recent stories with the same tag or section
ProductsBuildersMulti-sourceSelected01:23
vLLM Project Shows KV Cache and Kernel Optimizations at PyTorchCon 2026

vLLM now spans three hardware ecosystems, which reduces the lock-in that previously tied inference stacks to NVIDIA GPUs. Teams running self-hosted inference can benchmark vLLM against vendor-specific runtimes on AMD and TPU infrastructure. The KV cache management work matters most for serving long-context models where memory, not compute, is the bottleneck.

#vLLM#PyTorch#KV cache#inference
Event
官方账号PyTorch@PyTorch11 sourcesSource
How-toBuildersOfficialSelected00:28
vLLM push-based NIXL connector cuts TTFT in disaggregated inference

Shifting KV-cache transfer coordination off the critical path removes a hidden latency tax in prefill-decode disaggregation, and the 3x TTFT gain applies to any team serving LLMs at scale on vLLM. Practitioners running inference on AWS can adopt the NIXL connector on EC2 P5en with EFA networking directly from the guide. Compared with pull-based transfer, the push design needs no receiver-side polling, which is what unlocks the latency savings.

#vLLM#NIXL#KV cache#inference optimization
官方一手AWS Machine Learning Blog@Sunita NadampalliSource
How-toBuildersSelected21:24
Stanford CS336 Lecture 10 Covers LLM Inference Optimization

Percy Liang's closing argument reframes the KV cache as the structural root of Transformer inference inefficiency, pointing to architectures designed for inference from day one, such as SSMs, linear attention, and diffusion-based non-autoregressive generation. Engineers building or tuning inference stacks get a single coherent map from architecture to serving systems. The executable lecture notes with sympy-verified formulas make the material directly reproducible rather than slide-level theory.

#Stanford CS336#Percy Liang#LLM inference#KV cache
Event
shao__meng@shao__meng2 sourcesSource
PapersBuildersSelected22:20
Paper Tests Multi-Agent Systems With Persistent Memory on Long-Context Work

The finding suggests viral multi-agent memory demos may not transfer to standard evaluation setups, since benchmarks are structured in ways that neutralise recall benefits. Engineers deciding whether to add persistent memory to agent pipelines should weigh the near-zero measured accuracy gain against its efficiency advantages. The paper's benchmark-design critique is a useful checklist when validating similar architectures.

#multi-agent systems#persistent memory#RAG#long-context
elvis@omarsar0Source