长上下文推理的算力瓶颈是AI应用落地的关键障碍，做LLM推理优化或长文档处理的团队可以直接参考RTPurbo的稀疏化思路，无需从头训练模型。
阿里巴巴与南京大学联合发表论文，提出RTPurbo方法，通过轻量级适配将百万token预填充速度提升9.36倍（对比FlashAttention-2）。该方法发现训练好的全注意力模型已存在隐藏稀疏结构，无需重新训练。RTPurbo识别出少数需要远距离token的注意力头，其余头聚焦邻近文本，并使用16维索引器快速定位关键token。在长上下文基准和推理任务中，RTPurbo保持接近全注意力的精度，同时实现高达9.36倍加速。这证明长上下文推理中的浪费比表面看起来更有结构性。
New Alibaba + Nanjing Univ paper claims million-to…
New Alibaba + Nanjing Univ paper claims million-token prefill can be sped up 9.36X (compared against FlashAttention-2) with only lightweight adaptation
Shows standard LLMs can handle very long context faster by making attention selectively sparse.
The problem is that full attention gets very expensive when the input grows to hundreds of thousands or 1M tokens, because the model keeps comparing too many tokens with too many other tokens.
The paper’s claim is that a trained full-attention model already has a hidden sparse structure, so the model does not need to be rebuilt or trained from scratch.
RTPurbo uses that structure by finding the few attention heads that really need faraway tokens, while letting the other heads focus mostly on nearby text.
For those retrieval heads, it uses a small 16-dimensional token finder to guess which old tokens matter, then runs the real attention only on that selected set.
The authors tested this on long-context benchmarks and reasoning tasks, and RTPurbo kept accuracy close to full attention while reaching up to 9.36x faster prefill at 1M tokens and about 2x faster decoding.
RTPurbo's engineering rule: keep expensive long-context access only where it matters, and route the rest through a smaller search space.
The clever part is the 16-dimensional indexer.
It does not replace the model’s real attention computation; it acts like a cheap scout, finding likely useful tokens before the full representation is used on the selected set.
RTPurbo is not proof that every model can be safely sparsified this way.
But it is strong evidence that the waste in long-context inference is more structured than it looks.
----
Paper Link – arxiv. org/abs/2605.16928v1
Paper Title: "Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps"