这篇论文提供了Transformer注意力中LoRA秩选择的深入理论分析，对于理解LoRA在Transformer模型中的应用非常有帮助，特别是对于想要优化模型性能的工程师来说。
本文提供了一种针对Transformer注意力中低秩适应（LoRA）更新的秩选择的理论，针对每个LoRA秩提供了可达到的近似误差。证明了在目标注意力概率远离零的情况下，误差的下界与$ψ(\|d\|_2)$成比例，其中$d$是候选和目标注意力分数之间的差异。还提供了当候选分数保持在目标分数固定范围内时的目标-Fisher界限，以及当大部分概率质量集中在子集标记上的无限制下界。这些光谱界限描述了有限分数近似。最后，将分析扩展到融合多头LoRA和联合查询/键更新，揭示了秩共享和查询/键分解约束的影响。
How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention
Choosing the rank of a low-rank adaptation (LoRA) update is usually an empirical task. In this paper, we provide a task-dependent theory of the approximation error achievable at each LoRA rank for Transformer attention. We fix a pretrained attention head, a target attention function, and a distribution over inputs from the downstream task, and bound the smallest expected Kullback--Leibler (KL) error achievable by a rank-$r$ query LoRA update. When target attention probabilities are bounded away from zero, we prove a lower bound of the error proportional to $ψ(\|d\|_2)$, where $d$ is the difference between candidate and target attention scores and $ψ(t)=\min\{t^2,t\}$. We also prove an unconditional upper bound $\min\{\|d\|_2^2/4,\sqrt2\|d\|_2\}$. Under explicit realizability, geometry, and moment conditions, we then bound the best rank-$r$ error between an explicit multiple of $ψ(\sqrt{T_r})$ and $\min\{T_r/4,\sqrt{2T_r}\}$, where $T_r$ is the downstream-weighted tail energy of the target update. We also provide target-Fisher bounds when candidate scores remain within a fixed range of the target scores, and an unrestricted lower bound when a subset of tokens carries most of the probability mass. These spectral bounds describe finite-score approximation. We then construct explicit families in which softmax saturation makes the rank required to match the attention function strictly smaller than the rank required to match the finite logits. Finally, we extend the analysis to fused multi-head LoRA and joint query/key updates, exposing the effects of rank sharing and query/key factorization constraints.