GAAC：无损 KV 压缩方案，让长上下文推理更省内存
GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices
苹果和英伟达设备都能跑的无损 KV 压缩，内存省 26%、吞吐提 29%，做端侧长上下文推理的值得看看这套编解码思路。
论文提出 Group-Adaptive Additive Codec（GAAC），一种无损 KV cache 压缩与服务系统，利用 BF16 指数冗余在不损失精度的前提下压缩 KV。在五个测试模型上，GAAC 对 BF16 KV 达到 1.405-1.407 倍无损压缩；对 FP8 E4M3 缓存为 1.09-1.14 倍，E5M2 为 1.25-1.30 倍。在 NVIDIA DGX Spark 上，64K token 上下文的解码注意力加速 1.19-1.27 倍，端到端 W4A16 解码步骤加速 1.04-1.15 倍，服务态 KV 常驻内存减少约 26%，吞吐提升 13-29%。在 Apple M4 上注意力加速 1.02-1.25 倍，证明方案可移植到 NVIDIA 之外的硬件。
GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices
Weight-only, lossy quantization substantially reduces the storage and memory requirements for serving large language models (LLMs), but the growing key-value (KV) cache remains a bottleneck for long-context inference on personal devices. We present Group-Adaptive Additive Codec (GAAC), a lossless KV compression and serving system that exploits exponent redundancy to reduce storage and accelerate attention without introducing KV approximation. GAAC encodes BF16 exponents using calibrated per-channel base-plus-offset codes over contiguous ranges. GAAC adapts code widths across small groups while preserving KV values bit-for-bit. A fused attention kernel reconstructs KV tiles on the fly, avoiding full-cache decompression. Block-wise compression and fixed-capacity slots support incremental cache updates and paged serving. Across five tested models, GAAC achieves 1.405-1.407$\times$ lossless BF16 KV compression. We also study compression performance on low-precision tensors FP8 E4M3 caches ( 1.09-1.14$\times$) and E5M2 caches (1.25-1.30$\times$). On NVIDIA DGX Spark, GAAC accelerates decode attention at a 64K-token context by 1.19-1.27$\times$ and end-to-end W4A16 decode steps by 1.04-1.15$\times$. In serving, it reduces resident KV memory by approximately 26% and increases aggregate throughput by 13-29% under comparable KV-memory budgets. On Apple M4, GAAC achieves 1.02-1.25$\times$ attention speedups, demonstrating portability beyond NVIDIA GPUs.