SVD-Attention：用 SVD 分解诊断 Transformer 注意力头的低秩冗余
SVD-Attention: Efficiently Analyzing Attention Blocks in Transformers
arXiv 上一篇分析工具论文，用 SVD 拆开注意力头看哪些层真有冗余，还能和剪枝叠起来省 8-11% 推理计算。
SVD-Attention 是一种事后分析方法，对每个注意力头的 query-key 交互矩阵做奇异值分解，用谱幅值和校准集上的激活方差给奇异模式打分。在 7 个预训练模型和 5 类任务上，诊断显示冗余分布依赖任务类型：情感分类、图像分类和生成任务冗余集中在早中层，稠密预测和抽取式问答集中在后层。按诊断结果截断层最多损失 1.75 F1，随机截断同样数量的层最多损失 50 F1。该方法单独使用可减少 0.8-1.5% 推理 FLOPs，与权重剪枝组合后可减少 8-11%，任务质量变化在 0-4% 之间。
SVD-Attention: Efficiently Analyzing Attention Blocks in Transformers
Pretrained transformers spend a large share of their inference compute inside self-attention, yet the per-head query-key interaction is structurally low-rank by construction. We ask a diagnostic question: across layers and architectures, which heads actually use the full rank budget on real data, and which do not? We propose SVD-Attention, a post-hoc method that decomposes the per-head interaction matrix $M_h$ and scores every singular mode by both its spectral magnitude and its activation variance across tokens on a small calibration set. The score keeps the modes the head can amplify and that real data actually activates, and discards the rest; the resulting factorization is exact for the retained modes and needs no finetuning. Across seven pretrained models and five task types, the diagnostic finds a task-dependent pattern: redundancy sits in the early to middle layers for sentiment and image classification and for generation, and in the late layers for dense prediction and extractive question answering. It is also predictive: truncating the layers it flags costs at most $1.75$ F1, while truncating the same number of layers elsewhere in the same model costs up to $50$ F1. SVD-Attention is an analysis tool rather than a compression method: acting on it reduces inference FLOPs by $0.8$-$1.5\%$ on its own and by $8$-$11\%$ when composed with weight pruning, with $0$-$4\%$ change in task quality.