发现一个几乎零成本的训练技巧：缩小初始化尺度能大幅提升大模型推理能力。
论文发现缩小参数初始化尺度能持续改善大语言模型的预训练效果，在推理密集型任务上提升最为显著，同时识别出两种常见训练设置会抑制该优势。研究揭示了初始化尺度的关键平衡点，并发现小初始化驱动参数先凝聚为低复杂度结构再扩展为丰富表示。基于此提出γ初始化规则——将初始化范围作为可调旋钮，默认使用小初始化几乎不增加成本即可改善训练和推理。
Small Initialization Matters for Large Language Models
Large language models provide a tractable system for asking how intelligence itself emerges, rather than only how LLMs can be engineered. Although progress is usually attributed to scale, data and architecture, we show that parameter initialization is a gene-like determinant of training and, in particular, of model capacity. Reducing the initialization scale consistently improves pretraining, with the largest gains on reasoning-demanding tasks. We identify two widely used empirical settings that restrain the advantage of small initialization, and show how relaxing them restores favorable scaling. We further uncover a critical initialization that balances the reasoning and training. Mechanistically, small initialization drives a distinct developmental trajectory: parameters first condense into low-complexity structures and later expand into richer representations, giving concrete form to the idea that compression is intelligence. Token-level analyses show that the gains concentrate on non-trivial, context-constrained predictions rather than all tokens uniformly. These results motivate a simple $γ$-initialization rule: expose initialization rage as an explicit knob and use small initialization by default, an almost cost-free intervention that improves pretraining and strengthens reasoning across model scales.