Apple 研究：门控机制决定 State Space Model 的记忆与上下文学习能力
On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
Apple 机器学习团队拆解了 SSM 打不过 Transformer 的关键原因：门控机制怎么分配记忆和上下文学习。做 Mamba 这类线性架构的同学值得看看，改进方向写得比较具体。
Apple 研究团队发布论文《On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models》，指出 SSM 在上下文学习和精确检索任务上落后于 Transformers 的原因可以归结为门控机制的作用。SSM 以常数内存和线性计算复杂度进行序列建模，计算特性优于 Transformer，但在需要 in-context learning 的场景表现不佳，这限制了其在大规模语言建模中的采用。
论文通过分析现代循环架构中普遍存在的 gating 组件，解释了 SSM 在这些任务上成败的机制：门控决定了模型在长期记忆（memorization）与上下文内学习之间的分配方式。研究者展示了 SSM 的能力短板并非不可修复，而是可以通过调整门控结构来改善。
这项工作为 Mamba 等 SSM 架构的改进方向提供了理论依据，对开发选择线性注意力替代方案的团队有直接参考价值，也说明 Transformer 在检索类任务上的优势有明确的架构层面的解释。
On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
State Space Models (SSMs) have emerged as a compelling alternative to Transformers, enabling sequence modeling with constant memory and linear compute. Although SSMs exhibit reasonable performance and favorable computational characteristics, they continue to lag behind Transformers on tasks that require in-context learning and precise retrieval, slowing their adoption for large-scale language modeling. In this work, we demonstrate that both the success and failure of SSMs in these domains can be explained by studying the role of the gating mechanism, a prevalent component in modern recurrent…
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include SSM, Mamba, Transformer, 论文解读.
Primary source verification: published directly by Apple ML Research with first-party authority.
