Zyphra's ZHA hierarchical attention speeds up long-context models
The plug-in design means no retraining of the base model, so teams running open-source models could adopt ZHA by training or downloading a small index module. It matters for anyone serving long-context workloads, where KV-cache reads dominate cost and latency. Compared with MSA, the reported 12.7-point retrieval gain is the concrete benchmark to watch when evaluating sparse attention options.
Zyphra researchers developed ZHA, a hierarchical attention mechanism that organises a model's past context into a tree-shaped index resembling a book table of contents. At inference, the model follows the top-level index and reads only relevant fragments instead of rescanning all history. Existing open-source models need no retraining: a lightweight index module trained on 1 billion tokens can be plugged in. On a 256,000-token test covering retrieval, math reasoning and coding, a Qwen (通义千问) model with ZHA read only 1%–6% of history while matching near full-context scores, and beat the sparse attention tool MSA by 12.7 points on long-text retrieval accuracy. Zyphra estimates ZHA overtakes standard attention at 48,000 tokens and pulls clearly ahead beyond 1 million tokens.
ZHA（Zyphra分层注意力机制）：用树状目录索引帮AI大模型快速处理超长文本
现在的AI大模型在阅读长文章或进行长时间对话时，随着字数增加，需要反复从头核对所有历史内容，导致计算量成倍飙升、运行速度变慢。
为了解决这个问题，Zyphra研究团队开发了名为ZHA的分层注意力机制。这套方法把AI读过的历史内容层层打包，整理成类似书本目录的树状结构。每次AI需要回忆前面的信息时，会直接顺着顶层目录往下查找，只挑出跟当前问题相关的片段来读。现有的开源大模型不需要重新训练，只要需要额外用10亿token的数据训练一套轻量级的目录索引模块，接入就能使用。
改造完成后，AI在保持原有智商的前提下实现了大幅提速。在25.6万token的超长文本检索、数学推理和写代码测试中，接入ZHA的通义千问模型每次只需要读取1%到6%的历史记录，就能拿到和原本全文阅读几乎一样的分数，长文本检索准确率比同类稀疏注意力工具MSA高出12.7个百分点。按照Zyphra团队的测算，当文本长度达到4.8万token时，ZHA的速度开始超越传统模式；文本超过100万token后优势全面拉开，未来甚至能支撑AI处理上亿乃至十亿token的超长内容。
论文：https://t.co/7iCayZ85OM
