PAA-Bench 提出重新评估 VLM 视觉 token 剪枝：准确率和 token 数都不是好指标
Are We Evaluating VLM Token Pruning Right? Looking Beyond Accuracy and Token Count
论文发现现行的 VLM 剪枝评测方式有漏洞，模型可能靠文本捷径蒙混过关。作者做了个 PAA-Bench 框架和 PCS 指标，8 种方法实测后发现压缩更多 token 未必更快，想给剪枝方法做评测的可以看看。
一篇 arXiv 论文指出，现有 VLM 视觉 token 剪枝评估依赖聚合准确率和保留 token 数两个代理指标，存在缺陷：聚合准确率会掩盖剪枝方法是否真正识别关键视觉 token，模型可能利用文本捷径得分；相同的 token 预算因剪枝位置和方式不同，端到端成本差异很大。作者推出 PAA-Bench 评估框架，并提出 Random-Pass@k 衡量随机剪枝下样本仍答对的比例，以及 Preservation-Constrained Speedup（PCS）报告统一保留目标下的最大端到端加速。实验覆盖 8 种剪枝方法、3 个 VLM 骨干和 9 个基准，结果显示更强的 token 压缩不一定带来更大加速，效率排名随保留目标变化。
Are We Evaluating VLM Token Pruning Right? Looking Beyond Accuracy and Token Count
Visual token pruning aims to reduce inference cost while preserving the capabilities of the original VLM. Yet existing evaluations rely on aggregate task accuracy and retained token count as proxies for pruning quality and actual efficiency. We reveal two limitations: 1) Aggregate accuracy can obscure whether a pruning method truly identifies critical visual tokens; many examples remain solvable under arbitrary token removal and models can exploit textual shortcuts when visual evidence is degraded. 2) Retained token count is an unreliable proxy for efficiency, as identical token budgets can incur different end-to-end costs depending on where and how pruning is performed. Consequently, comparisons based on aggregate accuracy and retained token counts fail to recover the global performance-efficiency Pareto frontier across methods. We introduce PAA-Bench, a plug-and-play evaluation framework for existing benchmarks, to evaluate whether pruning succeeds through meaningful token selection and whether this success translates into real speedup. To determine when token choice actually matters, we propose Random-Pass@k to measure how often an example remains correct under random pruning, enabling comprehensive evaluation across different levels of selection difficulty. To compare efficiency fairly, we introduce Preservation-Constrained Speedup (PCS) to report the maximum end-to-end speedup each method achieves under a shared preservation target. Experiments across 8 methods, 3 VLM backbones, and 9 benchmarks show that stronger token compression does not necessarily yield greater speedup, method differences widen on examples where token selection matters, and efficiency rankings change with the preservation target.
