这篇论文把多模态预训练的内部机制拆开了，告诉你模态之间怎么互相影响、什么时候协同、什么时候拖后腿，还有省算力的训练配方，值得看看。
该论文通过合成和真实数据上的受控实验，揭示了多模态预训练中语言、视觉理解与视觉生成之间的非对称知识流。研究发现数据复杂度决定模态协同，共享注意力与归一化能促进协同，而早期联合训练优于晚期对齐。实验还发现“视觉懒惰”现象，即延迟融合会导致模型依赖语言先验。作者提出的高效配方仅用5%计算预算即可达到强生成性能，并训练了13.5B MoE模型在2T tokens上验证结论。
Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pretraining: (i) Knowledge Flow: We disentangle how language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry; (ii) Synergy vs. Competition: We show that data "complexity" largely determines whether modalities are synergistic, identify architectural choices that promote synergy: such as shared attention and normalization with modality-specific feed-forward layers, and find that these behaviors generalize across different visual tokenizer designs; (iii) Early Unification: Unifying modalities from the very early stages and training them jointly is shown to be more effective than late alignment or sequential training. This process uncovers a vision laziness phenomenon, where delayed integration leads models to rely on language priors; (iv) Recipes: We derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget. These core findings are subsequently validated at scale by training multiple 13.5B MoE models on 2T tokens. We hope this study provides a principled foundation for understanding and scaling multimodal pretraining.