线性注意力一直有长程遗忘问题，HOLA用一个小缓存就解决了，在32k长度上效果惊人。
HOLA论文提出为线性注意力添加海马体互补存储，保留delta-rule状态的压缩记忆并增加一个有限精确KV缓存，形成半参数测试时记忆。在340M参数、15B SlimPajama tokens训练下，WikiText困惑度从27.32降至22.92，低于全注意力Transformer++的26.88。在RULER needle recall测试中，HOLA在32k tokens（16倍训练长度）上保持稳健。
NEW paper worth reading. (bookmark it) The basic idea is to pair a compressive recurrent state wit...
NEW paper worth reading. (bookmark it) The basic idea is to pair a compressive recurrent state with a small exact memory, which helps to recover long-range recall without giving up the efficiency of linear attention. More on it below: Linear-attention and state-space models compress the whole prefix into a fixed-size state. That buys O(1) memory, but when many key-value associations compete, earlier facts get overwritten and needle recall degrades. HOLA gives linear attention a hippocampal complement. It keeps the usual delta-rule state as compressive memory and adds a bounded exact KV cache, forming a semiparametric test-time memory. The state models linearly compressible structure while the cache stores associations that should not be forced through it. The cache writes without a learned eviction module, keeping only tokens whose prediction residual was actually committed to the state. At 340M parameters on 15B SlimPajama tokens, HOLA lowers Wikitext perplexity from 27.32 to 22.92, below a full-attention Transformer++ at 26.88, and stays robust on RULER needle recall out to 32k tokens, 16x its training length. Paper: arxiv.org/abs/2607.02303 Learn to build effective AI agents in our academy: academy.dair.ai 💬 3 🔄 2 ❤️ 18 👀 2226 📊 7 ⚡
- arXiv cs.AI07-02 15:19原文