Matryoshka语言模型套件：嵌套训练省36%算力
Matryoshka Language Model Suites
想省训练算力又不想掉性能？这个嵌套训练法把三个模型塞一个架构里，直接省36%算力，还顺带加速推理，值得一看。
Matryoshka训练框架将500M、1.5B和3B三个子模型堆叠进单一嵌套架构，端到端训练。
相比独立训练基线，该套件在基准性能和困惑度上持平，但训练计算量减少36%。嵌套结构天然支持投机解码，吞吐量提升14-26%。
论文还消融了关键架构选择，为构建强Matryoshka套件提供指导。
Matryoshka Language Model Suites
Training a language model suite classically requires training each model separately and serving them independently. We improve both training and inference efficiency by stacking sub-models of increasing size into a single nested architecture trained end-to-end. This Matryoshka training framework reduces the total parameter count of the suite, enables low-cost distillation from the largest to all smaller sub-models at every training step, and is well-suited for speculative decoding as the draft model is contained within the verifier. We validate our approach by training a Matryoshka suite comprising 500M, 1.5B, and 3B sub-models. Our suite is on par with independently trained baselines on benchmark performance and validation and out-of-domain perplexities, while using 36% less training compute and improving the throughput of speculative decoding by 14-26%. We also ablate key architectural choices, offering guidance for building strong Matryoshka LM suites.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include Matryoshka, 语言模型, 训练效率, 投机解码.
Source attribution: curated from arXiv cs.AI (@Nathan Godey, Yoav Artzi).
