训练 LLM 的团队终于有了一个不改变模型、不调优化器的加速方案——直接插进去就能省 2.5 倍时间，做预训练或资源受限的开发者值得一试。
Nous Research 发表论文提出 Token Superposition Training，一种无需改变模型架构或优化器的即插即用方法，可将 LLM 预训练时间缩短最多 2.5 倍。该方法将连续 token 分组并平均其嵌入，然后联合预测下一组，使用多热交叉熵损失，并在训练中期（20-40% 后）切换回标准 next-token prediction。最终模型权重与常规训练一致，在 270M 到 10B MoE 规模上均优于基线损失和下游评估。这意味着训练成本大幅降低，且不影响最终模型质量。
What if you could train LLMs 2-3x faster without c…
What if you could train LLMs 2-3x faster without changing the final model at all?
Most of that money goes into processing one token at a time, billions of times over. Nous Research published a paper introducing Token Superposition Training. It's a drop-in method that cuts pretraining time by up to 2.5x. Here's how it works: > Group contiguous tokens into bags > Average their embeddings together > Predict the next bag jointly > Use multi-hot cross-entropy as loss > Revert to normal training midway The switch happens after the first 20-40% of training. From there, it finishes with standard next-token prediction. The final weights match a conventionally trained model. No changes to the optimizer, tokenizer, data, or architecture are required. The team tested it at 270M, 600M, 3B, and a 10B mixture-of-experts scale. It consistently beats baseline loss and downstream evals.