LDM：用递归最小二乘做跨层记忆检索，困惑度优于 mHC 与 AttnRes
Depth Memory
LDM 把跨层信息传递改成可检索的线性记忆，400M 与 1.5B 规模困惑度都低于 mHC 和 AttnRes。
残差连接通过逐层累加输出在网络中传递信息，近期提出的 mHC 与 AttnRes 改用更具表达力的深度聚合机制来提升大规模语言建模表现。论文将跨层表示混合视为跨层存储与检索信息的问题，提出统一框架 Depth Memory（DM），把标准 PreLN transformer、mHC 和 AttnRes 统一描述为携带表示穿过网络的流。在此视角上提出 Linear Depth Memory（LDM），把跨深度信息检索形式化为用递归最小二乘求解的回归问题，在深度方向传播残差流、线性联想记忆流和逆协方差流三条流。配套的硬件感知实现降低峰值激活内存与内存流量，LDM 在 400M 和 1.5B 参数规模上困惑度均低于 PreLN、mHC、AttnRes 基线，零样本精度也具竞争力。
Depth Memory
Residual connections propagate information through a network by additively accumulating layer outputs. Recently proposed alternatives, such as Manifold-Constrained Hyper-Connections (mHC) and Attention Residuals (AttnRes), instead use more expressive depth aggregation mechanisms that improve language modeling performance at scale. In this work, we consider the mixing of representations over depth as a problem of storing and retrieving information across layers. Formalizing this perspective, we introduce Depth Memory (DM), a unified framework that describes the standard pre-layer-norm (PreLN) transformer, mHC, and AttnRes in terms of streams that carry representations through the network. One such stream, for example, could be an explicit memory stream that is queried to enrich the input to every layer, and whose output is subsequently written back into the memory for use by later layers. For AttnRes, this memory stream is a growing cache of explicit layer representations. For mHC, it is an expanded residual stream. Building on this view, we propose Linear Depth Memory (LDM), which formulates information retrieval across depth as a regression problem solved using recursive least squares. The resulting architecture propagates three streams across depth: the standard residual stream, a linear associative memory stream, and an inverse covariance stream. For high-throughput training and inference, we propose a hardware-aware implementation of LDM that reduces peak activation memory and memory traffic. LDM consistently achieves the lowest perplexities among PreLN, mHC and AttnRes baselines at the 400M and 1.5B parameter scales, alongside competitive zero-shot accuracy.
