谷歌Titans团队的新方法用记忆缓存把RNN长上下文召回从21提到32，算力还远低于Transformer。
谷歌Titans团队提出记忆缓存方法，将RNN与Transformer统一为同一框架下的两种设置。该方法通过保存记忆检查点，使RNN的有效记忆随输入长度增长，计算复杂度为O(NL)，介于RNN的O(L)和Transformer的O(L²)之间。在16k针堆测试中，Titans模型的召回得分从21提升至32。该方法可后训练接入，让模型处理超出训练长度的上下文。纯Transformer在原始召回精度上仍占优，但记忆缓存在极低算力下大幅缩小了这一差距。
not saying this is right but a single discovery like this could undermine the need for megascale dat...
not saying this is right but a single discovery like this could undermine the need for megascale date centers Guri Saroy @HeyGurisaroy Google just proved the Transformer is the most expensive version of a much bigger idea. Here's the setup. There are two ways for a model to remember a long context. → RNNs crush everything into one fixed memory. cheap, but they forget. → Transformers keep every token forever. sharp, but the cost grows quadratically. twice the context, four times the compute. For years those felt like two different species. This paper shows they're the same thing at two settings of one dial. The idea is called Memory Caching. Instead of overwriting your memory as you read, you snapshot it. Every so often you save a checkpoint of the memory state and keep it around. When a new token needs to look something up, it can reach back into every saved memory, not just the latest one. So the RNN stops forgetting. Its effective memory now grows with the length of the input. The complexity lands at O(NL). that sits right between the RNN's O(L) and the Transformer's O(L²). N is the dial. Now the part that made me stop. Turn the dial all the way up. Make every single token its own memory checkpoint. Save all of them. You just rebuilt attention. That's what a Transformer is. Memory caching with the segment size set to 1. Every token cached, nothing compressed, maximum cost. The setting where you pay full price. Attention was never the fundamental thing. It's the far end of a spectrum. What that buys you in practice: → Bolt it onto Titans, deep linear attention, or plain linear attention. all of them get better. → Long-context recall climbs hard. on a 16k needle-in-a-haystack test, Titans went from 21 to 32. → It stays subquadratic, so it's far cheaper than a Transformer as context gets long. → You can even bolt it on after training and the model reaches longer contexts than it was trained on. The honest ceiling: a pure Transformer still wins on raw recall accuracy. This closes the gap, it doesn't erase it. But it closes it while spending a fraction of the compute. From the same Google team that built Titans. 🔗 View Quoted Tweet 💬 0 🔄 0 ❤️ 1 👀 441 ⚡