Nous Research 提出 Token Superposition Training，LLM 预训练速度提升 2.5 倍
训练 LLM 的团队终于有了一个不改变模型、不调优化器的加速方案——直接插进去就能省 2.5 倍时间，做预训练或资源受限的开发者值得一试。
Nous Research 发表论文提出 Token Superposition Training，一种无需改变模型架构或优化器的即插即用方法，可将 LLM 预训练时间缩短最多 2.
5 倍。该方法将连续 token 分组并平均其嵌入，然后联合预测下一组，使用多热交叉熵损失，并在训练中期（20-40% 后）切换回标准 next-token prediction。
最终模型权重与常规训练一致，在 270M 到 10B MoE 规模上均优于基线损失和下游评估。这意味着训练成本大幅降低，且不影响最终模型质量。
What if you could train LLMs 2-3x faster without changing the final model at all?
Most of that money goes into processing one token at a time, billions of times over. Nous Research published a paper introducing Token Superposition Training. It's a drop-in method that cuts pretraining time by up to 2.5x. Here's how it works: > Group contiguous tokens into bags > Average their embeddings together > Predict the next bag jointly > Use multi-hot cross-entropy as loss > Revert to normal training midway The switch happens after the first 20-40% of training. From there, it finishes with standard next-token prediction. The final weights match a conventionally trained model. No changes to the optimizer, tokenizer, data, or architecture are required. The team tested it at 270M, 600M, 3B, and a 10B mixture-of-experts scale. It consistently beats baseline loss and downstream evals.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include 预训练加速, Token Superposition Training, Nous Research, LLM训练.
Source attribution: curated from AlphaSignal (@AlphaSignalAI).