随机化设计为KV-Cache驱逐提供可证明误差证书

Error Certificates for KV-Cache Eviction via Randomized Design

精选理由

一篇论文证明了随机化KV-Cache驱逐能提供误差证书,覆盖率达97%。如果你关心推理加速中的精度损失量化,这方法很实用。

AI 摘要

确定性KV-Cache驱逐保留top-k token,论文证明该方法无法估计被丢弃内容导致的注意力输出误差。随机化驱逐通过泊松采样尾部并应用Hájek校正,在softmax中实现,保留集上的方差估计可作为每步误差证书,经验覆盖率达0.97且无精度损失。在真实工作负载上预注册七个主张,其中三个被否定:问题感知驱逐在25-50%预算下近乎免费;输出对数概率比证书更好预测失败;证书门控预算升级无增益。留存结果为:证书可分离缓存引发与固有失败(AUC 0.73-0.75,输出置信度仅0.47-0.54),并比随机或置信度门控更优地安排重计算。

原文 · arXiv cs.AI

Error Certificates for KV-Cache Eviction via Randomized Design

Deterministic KV-cache eviction keeps the top-$k$ tokens under an importance score and deletes the rest. We prove that this design cannot know what it destroyed: evicted values can be altered so that everything the serving system retains is unchanged while the true attention-output error grows arbitrarily, so no serving-time estimator of that error is consistent. Randomized eviction restores identifiability. With a Poisson-sampled tail at known inclusion probabilities, one logit offset performs the Hájek correction inside the softmax, and a survey-sampling variance estimator over the retained set becomes a per-step error certificate with 0.97 empirical coverage at no accuracy cost. On real workloads we pre-registered seven claims and lost three: question-aware eviction at 25--50\% budgets is nearly free; output log-probability predicts failure better than the certificate; certificate-gated budget escalation adds nothing. What survives is attribution: the certificate separates cache-induced from inherent failures (AUC 0.73--0.75, against 0.47--0.54 for output confidence) and schedules recomputation better than random or confidence gating. Randomization buys attribution, not prediction.