这篇论文提出了InfoKV，用信息熵改进KV缓存压缩，在Llama-3.1和DeepSeek-R1上比传统注意力方法效果更好，适合关注长推理效率的人。
论文提出InfoKV，一种熵感知的KV缓存压缩框架，通过结合token级预测不确定性与层表示演化来估计重要性。在Llama-3.1、Llama-3.2和DeepSeek-R1上的长上下文推理基准实验中，InfoKV在预填充和解码阶段持续优于现有基于注意力的压缩方法。该方法引入Forward Influence度量，发现高不确定性token对远距离上下文影响更大。
Information-Aware KV Cache Compression for Long Reasoning
Reasoning capability has advanced rapidly in large language models (LLMs), leading to an increasing size of key-value (KV) cache in both prefilling and decoding stages. Existing KV cache compression methods mainly rely on attention weights to estimate token importance. While attention effectively captures contextual relevance, it overlooks complementary information-theoretic signals related to predictive uncertainty and token informativeness. In this paper, we revisit token importance from a forward-looking perspective and introduce \textit{Forward Influence}, a metric that measures how compressed tokens affect future contexts. Our analysis reveals that tokens selected by attention scores mainly influence nearby contexts, whereas tokens associated with high predictive uncertainty exhibit substantially stronger influence on distant future contexts. Based on the observation, we propose \textbf{InfoKV}, an entropy-aware KV cache compression framework that incorporates information-theoretic signals. It combines token-level predictive uncertainty with layer-wise representation evolution and integrates the resulting entropy scores with attention scores during reasoning. Experiments on long-context reasoning benchmarks with Llama-3.1, Llama-3.2, and DeepSeek-R1 demonstrate that InfoKV consistently outperforms existing attention-based KV compression methods in both long prefilling and decoding scenarios.