这篇论文提出HiKV，用两阶段KV缓存压缩加专用硬件，7.95倍加速、90%省电，精度几乎不掉。做长上下文LLM推理加速的可以看看。
HiKV是一种算法-硬件协同设计的KV缓存压缩方法，通过两阶段重要性感知实现高压缩率：第一阶段在固定预算内驱逐不重要的token，第二阶段对每个保留token仅加载显著元素。基于可重构重要性排序器的专用加速器统一两阶段的数据通路，仅增加8%系统面积。在代表性LLM上，HiKV实现高达7.95倍加速和90%能耗降低，精度损失小于1%。在等精度约束下，相比其它重要性方法额外减少1.82~4.87倍外部内存访问。
HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding
With the rapid adoption of long-context large language models (LLMs), the continuously growing KV cache during decoding has become the critical memory bottleneck. To tackle this challenge, we propose HiKV, a novel algorithm-hardware co-design that exploits KV cache redundancy through hierarchical importance awareness. Algorithmically, HiKV compresses the KV cache at two granularities: Stage I evicts unimportant tokens within a fixed budget, and Stage II further loads only the significant elements of each retained token, reaching compression ratios unattainable at a single granularity. Architecturally, we develop a dedicated accelerator centered on a reconfigurable importance sorter that switches between the distinct sorting datapaths each stage requires, unifying the two-stage acceleration in one circuit with minimal overhead. Evaluated on representative LLMs, HiKV achieves up to 7.95x speedup and 90% energy reduction in the attention computation over the vanilla KV cache baseline within negligible 1% accuracy loss. Under iso-accuracy constraints, HiKV outperforms state-of-the-art importance-based methods by achieving an additional 1.82~4.87x reduction in external memory accesses. These benefits are enabled by specialized hardware components that add only 8% to the system area.