tree-grams:层次化 n-gram 模型成为可控的语言近似器
What's in a Tree ? Hierarchical $n$-gram models as controlled approximators of language
搞理论或调参的人可以看看这篇:tree-grams 让你能像拧旋钮一样控制语言模型的上下文长度和容量,用它生成可控数据做 μP 超参迁移实验,比直接在大语料上试更稳。还能剪掉 95% 的拓扑不损性能,思路对构建小型可解释模型也有启发。
一篇新论文提出 tree-grams,一种自回归生成式 n-gram 模型,通过阶数为 2 和 3 的张量层次化组合来构建条件概率张量 p(x_n | x_{1:n-1})。超参数 n 控制最大上下文窗口,秩超参数 d 控制模型容量,形成一个可解释、可系统扩展的语言近似器家族。
实验显示 tree-grams 优于现有 n-gram 基线,同时与深度 Transformer 语言模型保持竞争力。由于可以显式控制视野和容量,tree-grams 能被用作受控合成数据生成器:论文证明当用中等 n 和 d 的 tree-grams 生成更简单的数据来调参较窄模型时,μP 下的超参数迁移更加稳健;熵和有效相关范围则决定微调后的性能差距。
当允许在候选拓扑之间路由时,tree-grams 出现了涌现稀疏性,可以剪枝多达 95% 的拓扑而性能没有显著变化。
What's in a Tree ? Hierarchical $n$-gram models as controlled approximators of language
Hierarchical models have been proposed as synthetic settings for studying language. However, their relevance to realistic language corpora remains underexplored. In this work, we introduce tree-grams, autoregressive generative $n$-gram models that build the conditional probability tensor $p(x_n \mid x_{1:n-1})$ through a hierarchical composition of order-2 and order-3 tensors. $n$ controls the maximal context window, whereas a rank hyperparameter $d$ controls model capacity, yielding an interpretable and systematically scalable family of approximators of language. We show how tree-grams outperform existing $n$-gram baselines, while remaining competitive with deep transformer-based language models. We demonstrate the utility of such controlled approximations by using tree-grams as generative models. In particular, we show that hyperparameter transfer under $μ$P can be made more robust when narrower models are tuned on simpler data, generated by tree-grams with moderate $n$ and $d$. In addition, we show that entropy and effective range of correlations control performance gap after fine-tuning. This is because tree-grams allow us to explicitly control horizon and capacity. When allowed to route across candidate topologies, we observe emergent sparsity in tree-grams. Such concentrated routing permits pruning upto 95\% of topologies without significant change in performance.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include n-gram, Transformer, μP, language models.
Source attribution: curated from arXiv cs.AI (@Aritra Das, Francesco Cagnetta, Darshil Doshi, Victor V. Albert, Maissam Barkeshli, Andrey Gromov).