Papers

Chorus:共享n-gram嵌入表配合逐层适配,体积缩小87%

Chorus: Shared N-gram Embeddings with Layer-wise Adaptation

Why we picked it

DeepSeek 的 Engram 刚带火 n-gram 嵌入表,这篇论文就给了个更省内存的方案:一张共享表加逐层投影,1.2B 参数打出 9.6B 的效果。做预训练或研究架构的人可以看看实现思路,配 Ember 优化器内存还能再省 99.2%。

arXiv 论文提出 Chorus,一种新的 n-gram 嵌入表设计。核心成绩是:1.2B 参数的 Chorus 表在文本预测上超过 FactorEngram 和 DeepSeek Engram 用 9.6B 参数达到的效果,表体积缩小 87%,并且在 20M 到 9.6B 全部表尺寸区间上胜过 Engram、TN-gram 和 FactorEngram。

Body

技术机制上,现有做法要么每层单独维护一张 n-gram 表(内存开销大、层间无法共享知识),要么全网共用一张表(所有层拿到相同嵌入,缺少逐层偏好,实际表现差)。Chorus 采用单一共享表,但每层通过基于当前隐藏状态计算的缩放因子对嵌入分量重新加权,再经过该层自己的投影矩阵读出,相当于共享存储加逐层适配。

论文还验证了与内存优化器 Ember 的兼容性:相比 AdamW,Ember 在表优化器上节省 99.2% 内存且不降低训练性能。两者叠加后,n-gram 嵌入表沿表尺寸方向的扩展在内存层面变得可行,对参数预算紧张的开源模型训练团队有直接参考价值。

Original · arXiv: DeepSeek

Chorus: Shared N-gram Embeddings with Layer-wise Adaptation

Language models can gain capacity at little computational cost from n-gram embedding tables: large tables of learned vectors for short sequences of consecutive tokens, which are looked up and added to the model's input or to its intermediate layers. Recent designs commonly use a separate n-gram embedding table for each layer, which makes the tables costly in memory and prevents layers from sharing what they learn. Sharing a single table, however, gives every layer the same embedding and leaves no room for layer-wise preferences, and it performs poorly in practice. We propose Chorus, in which one set of n-gram embedding tables serves the whole network and every layer reads it in its own way. Each layer maps the refined embedding through its own projection layer, after rescaling its components by factors computed from its current hidden state. Chorus matches the quality of existing n-gram methods with 87% smaller tables: with 1.2B table parameters it predicts text better than FactorEngram and DeepSeek's Engram do with 9.6B. It outperforms Engram, TN-gram and FactorEngram in every setting where we compared them, across table sizes from 20M to 9.6B parameters. We also show that the recently proposed memory-efficient optimizer Ember works well with Chorus, saving 99.2% of table-optimizer memory compared with AdamW without degrading training performance. These results enable further scaling of n-gram embedding tables along the table-size axis by reducing memory use and allowing shared knowledge to be reused across layers.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Chorus, Engram, DeepSeek, open-source models.

Primary source verification: published directly by arXiv: DeepSeek with first-party authority.