RaBitQCache用旋转二进制量化KV缓存，比现有稀疏注意力方法更快更省内存，还能自适应调整预算。
RaBitQCache利用随机旋转二进制量化和二进制INT4算术高效估计注意力权重。该方法的代理分数是无偏估计量且具有可证误差界，支持自适应Top-p检索动态调整令牌预算。硬件感知系统通过异步流水线和惰性更新隐藏开销。实验表明，相比现有基线，RaBitQCache显著加速推理并减少内存I/O，同时保持生成质量。代码已在GitHub开源。
RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference
Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased. To address these limitations, we propose RaBitQCache, a novel sparse attention framework that utilizes randomized rotated binary quantization and high-throughput binary-INT4 arithmetic to efficiently estimate attention weights. Our proxy score serves as an unbiased estimator with a proven error bound, enabling adaptive Top-p retrieval that dynamically adjusts the token budget based on actual attention sparsity. We further implement a hardware-aware system with asynchronous pipelining and lazy updates to mask overhead. Evaluations demonstrate that RaBitQCache significantly accelerates inference and reduces memory I/O while preserving generation quality compared to state-of-the-art baselines. Code is available at https://github.com/Sakuraaa0/RaBitQCache.git.