利用注意力偏置为 Transformer 剪枝降低计算成本
Exploiting Biases in Attention for Transformers
给 token 打个影响力分数就能剪掉大块注意力计算，FLOPs 和延迟都降，代码也开源了，做视觉模型部署的可以看看。
EIDOSLAB 在 arXiv 发布论文，分析视觉 Transformer 中注意力矩阵各列对最终计算的重要性差异，发现该现象随网络深度加深而更明显。基于此提出在 query-key 点积中加入输入相关的偏置项，让每个 token 获得一个标量值作为其预期影响力的代理指标。该指标可用于剪枝注意力图，在多个规模的 Transformer 模型上同时降低 FLOPs 和推理延迟，同时保留大部分分类准确率。代码和模型已在 GitHub 上开源。
Exploiting Biases in Attention for Transformers
Attention is known to be sparse in vision transformers. In particular, we analyze the unequal relevance of attention-matrix columns to the final computation and show that this phenomenon becomes increasingly pronounced with network depth. This can be interpreted as saying that tokens have an absolute importance for the attention process, independently of queries. Inspired by this, we propose a reformulation of the attention formula that makes this phenomenon explicit by adding an input-dependent bias to the query-key dot product. We show that this new term gives each token a scalar value indicating its anticipated influence on other tokens during the attention process, and can thus serve as a proxy for pruning attention maps. Across multiple transformer model sizes, this allows us to substantially reduce the computational cost of attention, in terms of both FLOPs and inference latency, while retaining most of the original classification accuracy. We publicly released the code and models at https://github.com/EIDOSLAB/AttentionWithBias.
