这篇论文把稀疏和低秩两种思路拧在一起，解决Transformer的长序列瓶颈。不用算完整的注意力矩阵，就能同时抓住局部和全局信息。
ELSAA提出一种结合稀疏和低秩分支的注意力近似方法，避免显式计算完整的N×N注意力矩阵。稀疏分支捕捉高相似度交互，低秩分支压缩全局信息，并通过分母感知融合项平衡两者。该方法旨在支持更长上下文训练，同时保留token级交互和上下文混合能力。在多个长序列基准上验证了其效率与效果。
ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers
The quadratic $N\times N$ attention score matrix remains a central obstacle to extending Transformers to longer input lengths. Existing efficient attention methods usually reduce this bottleneck by either imposing sparsity, so that each query attends to only a small subset of keys, or by using low-rank/kernel sketches, so that global interactions are compressed into a lower-dimensional representation. We propose \emph{ELSAA}, an efficient low-rank and sparse approximation of attention. Importantly, ELSAA does \emph{not} decompose the learned projection or output matrices of the Transformer into sparse and low-rank factors. Instead, after dense projections produce $Q,K,V$, ELSAA approximates the induced attention score operator itself: a sparse branch captures selected high-similarity interactions, while a low-rank branch summarizes diffuse global interactions. Since the two branches can be normalized over supports with very different denominator mass, ELSAA introduces a denominator-aware fusion term that scales the sparse branch according to its estimated attention mass relative to the low-rank branch. This gives a practical framework for constructing low-rank and sparse attention outputs without materializing the full quadratic score matrix, aiming to enable longer-context training while preserving both sharp token-level interactions and broad contextual mixing.