做长上下文推理或 agent 工作流的开发者，终于有了一个能直接部署的稀疏注意力方案——MSA 在 109B 模型上实现 28 倍计算缩减，且内核已开源，值得立刻试跑。
MiniMax 提出了一种名为 MiniMax Sparse Attention (MSA) 的块级稀疏注意力机制，旨在解决大语言模型在超长上下文（百万级 token）下的计算瓶颈。MSA 基于分组查询注意力（GQA），通过轻量级索引分支对键值块进行评分，并为每个 GQA 组独立选择 Top-k 子集，实现高效的组级稀疏检索。在 109B 参数的多模态模型上，MSA 在 1M 上下文长度下将每 token 注意力计算量减少 28.4 倍，并在 H800 GPU 上实现 14.2 倍预填充和 7.6 倍解码加速。该方法的推理内核已开源，同时发布了基于 MSA 的生产级多模态模型。
MiniMax Sparse Attention
Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale. We introduce MiniMax Sparse Attention (MSA), a blockwise sparse attention built upon Grouped Query Attention (GQA). A lightweight Index Branch scores key-value blocks and independently selects a Top-k subset for each GQA group, enabling group-specific sparse retrieval while maintaining efficient block-level execution; the Main Branch then performs exact block-sparse attention over only the selected blocks. Designed around a principle of simplicity and scalability, MSA is deliberately streamlined, making it straightforward to deploy efficiently across a broad range of GPUs. To translate sparsity into practical speedups, we co-design MSA with a GPU execution path that uses exp-free Top-k selection and KV-outer sparse attention to improve tensor-core utilization under block-granular access. On a 109B-parameter model with native multimodal training, MSA performs on par with GQA while reducing per-token attention compute by 28.4x at 1M context. Paired with our co-designed kernel, MSA achieves 14.2x prefill and 7.6x decoding wall-clock speedups on H800. Our inference kernel is available at: https://github.com/MiniMax-AI/MSA. A production-grade natively multimodal model powered by MSA has been publicly released at: https://huggingface.co/MiniMaxAI/MiniMax-M3.