Latent Sparse Attention：让超长上下文注意力计算提速 72 倍
Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention
超长上下文跑不动？这篇论文用分层聚类做稀疏注意力，10M token 下比全注意力快 72 倍还不掉点，做长上下文的值得看看。
arXiv 论文提出 Latent Sparse Attention（LSA），一种在超过 1M token 的超长上下文下保持精度的稀疏注意力方法。LSA 将 key 在学习到的隐空间中分层聚类，query 只对聚类层级打分，无需逐一计算每个 key。在 GLM-5.3-FLASH（QWEN-3.8-27B）上处理 10M token 时，D-LSA 的 prefill 比原稀疏注意力快 22.7 倍，decode 比全注意力快 42.5 倍，精度基本持平。N-LSA 变体在 128k token 下以 1.6 倍的每层训练加速达到与全注意力相同的精度。
Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention
Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows. However, at ultra-long context (above 1M tokens), full attention is prohibitively expensive. A promising approach is sparse attention, in which a dedicated indexer selects a subset of keys for the attention. Yet current methods of indexing either still have quadratic cost in context size or lose accuracy at ultra-long context scale. We address this with Latent Sparse Attention (LSA), an efficient attention method that maintains accuracy at ultra-long context. In LSA, keys are hierarchically clustered in a learned latent space, and queries score the hierarchy of clusters instead of scoring every key, making prefill, decode and training faster at long context. We propose two variants, Distilled-LSA (D-LSA), which can be applied to pre-trained models via distillation, and Native-LSA (N-LSA), which is trained natively on the next-token loss alone. On GLM-5.3-FLASH (QWEN-3.8-27B) at 10M tokens, D-LSA speeds up one attention layer by 22.7x (72.4x) in prefill and 4.6x (42.5x) in decode over its original sparse attention (full attention), while keeping similar accuracy across a variety of tasks. N-LSA matches the accuracy of full attention with a training speedup of 1.6x per layer at 128k tokens. LSA offers practical and efficient attention for models to benefit from ultra-long context.