Latent Lattice语言模型:用持久横向潜态流降低困惑度
Latent Lattice Language Models
这篇论文把"加大每步计算状态"当成独立扩展维度,而不只是加深层数或加残差连接。406M 参数就在 FineWeb-Edu 20B 上把困惑度从 15.41 压到 14.40,还能和 hyper-connections、Mamba-2 这类已有架构叠加。做模型架构或研究线性注意力的开发者可以看看,思路可能影响下一代小模型设计。
arXiv 论文 2611.21608 提出 Lattice of Latents 架构,在逐层残差流之外引入一组沿时间方向循环的横向潜态流,增加模型每步维护的活跃计算状态数量。这些潜态直接调制纵向流的注意力、KV-cache 写入、MLP 计算和解码,与 hyper-connections、mHC 等只增加或重路由残差通路的方法不同,也与把循环状态仅用作序列混合内部记忆的 Transformer-SSM 混合架构不同。
实验方面,在 20B FineWeb-Edu tokens 上训练的 406M 参数 Lattice 模型,验证困惑度从参数匹配 Transformer 的 15.41 降至 14.40;与 hyper-connections 和 input residuals 组合后进一步降至 13.61。在 Gated DeltaNet、Mamba-2、Triadic GDN 和 GDN-Mamba-2 混合架构上,Lattice 在每个测试的原生状态规模都改善困惑度,Lattice-GDN 变体超过原生状态大 4 倍的纯基线。
Latent Lattice Language Models
Language models typically increase per-token computation through deeper or looped layers and latent reasoning steps, while richer residual connections improve information flow across layers. We instead increase computational capacity: the number of active computational states a model maintains. We introduce the Lattice of Latents, a set of per-layer horizontal latent streams that run recurrently along time, alongside the vertical residual stream that flows through depth. Like hyper-connections, mHC, and attention or value residuals, the Lattice enhances the residual stream, but rather than adding or rerouting residual pathways through depth, it provides companion computational states that persist across tokens. Unlike hybrid Transformer-SSM architectures, whose recurrent states primarily serve as internal memory for sequence mixing, these latents directly condition the vertical stream's attention, KV-cache writes, MLP computation, and decoding. On Transformers trained on 20B FineWeb-Edu tokens, a 406M-parameter Lattice model lowers validation perplexity from 15.41 to 14.40 over a parameter-matched Transformer. These gains extend across architectural variants. The Lattice composes with prior residual-stream enhancements: with hyper-connections and input residuals, it lowers validation perplexity from 15.41 (base Transformer) and 14.22 (Transformer with the same enhancements) to 13.61. On Gated DeltaNet, Mamba-2, Triadic GDN, and GDN-Mamba-2 hybrids, it improves perplexity at every tested native state size, with Lattice-GDN and Lattice-Triadic-GDN variants outperforming pure baselines with 4x larger native states. These results identify persistent per-layer state as a complementary axis for scaling language-model computation.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Lattice of Latents, Mamba-2, Gated DeltaNet, Transformer.
Source attribution: curated from arXiv cs.LG (@Shivam Duggal, Zongze Wu, William T. Freeman, Antonio Torralba, Richard Zhang, Eli Shechtman).