Sakana AI 提出 Continuous Memory Machine 双记忆矩阵架构
Sakana AI 给循环模型装了两块记忆，一快一慢，在记忆和迷宫任务上把 LSTM、CTM 这些老基线都超了，做架构研究的可以看看。
Sakana AI 发布论文介绍 Continuous Memory Machine（CMM），为循环模型引入短期与长期两个记忆矩阵，解决单一隐藏向量中计算与存储互相争抢空间的问题。短期记忆追踪最近的神经元活动，长期记忆存放后续步骤需要的信息，由 Transformer 在每一步读写。CMM 在复制、关联回忆、排序、少样本回归和迷宫求解任务上超过 LSTM、DNC、RMC 和 CTM 基线，并能泛化到比以往记忆增强网络更长的输入。注意力图显示模型只在算法类和上下文类任务调用长期记忆，无关任务则跳过。
Interesting paper from Sakana AI on memory for recurrent models.
Recurrent models are good at state tracking, but they usually keep everything in one hidden vector, so short-term computation and long-term storage compete for the same space.
The Continuous Memory Machine gives the model two memory matrices. One short-term memory tracks recent neuron activity, and the other long-term memory stores information for later steps. A Transformer reads and writes both at every step.
It builds on Sakana's Continuous Thought Machine and beats LSTM, DNC, RMC, and CTM baselines on copy, associative recall, sorting, few-shot regression, and maze solving. It also generalizes to longer inputs than earlier memory-augmented networks.
The attention maps show the model uses long-term memory for algorithmic and in-context tasks and skips it when the task does not need it.
Paper: https://t.co/89qZ1pbDTE
