多模态预训练物理学：知识流、模态协同、早期统一与配方
Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
这篇论文把多模态预训练的内部机制拆开了，告诉你模态之间怎么互相影响、什么时候协同、什么时候拖后腿，还有省算力的训练配方，值得看看。
该论文通过合成和真实数据上的受控实验，揭示了多模态预训练中语言、视觉理解与视觉生成之间的非对称知识流。研究发现数据复杂度决定模态协同，共享注意力与归一化能促进协同，而早期联合训练优于晚期对齐。
实验还发现“视觉懒惰”现象，即延迟融合会导致模型依赖语言先验。作者提出的高效配方仅用5%计算预算即可达到强生成性能，并训练了13.
5B MoE模型在2T tokens上验证结论。
Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes
Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pretraining: (i) Knowledge Flow: We disentangle how language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry; (ii) Synergy vs. Competition: We show that data "complexity" largely determines whether modalities are synergistic, identify architectural choices that promote synergy: such as shared attention and normalization with modality-specific feed-forward layers, and find that these behaviors generalize across different visual tokenizer designs; (iii) Early Unification: Unifying modalities from the very early stages and training them jointly is shown to be more effective than late alignment or sequential training. This process uncovers a vision laziness phenomenon, where delayed integration leads models to rely on language priors; (iv) Recipes: We derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget. These core findings are subsequently validated at scale by training multiple 13.5B MoE models on 2T tokens. We hope this study provides a principled foundation for understanding and scaling multimodal pretraining.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include 多模态, 预训练, MoE, 知识流.
Source attribution: curated from arXiv cs.LG (@Junlin Han, Shengbang Tong, David Fan, Minghao Chen, Philip Torr, Filippos Kokkinos, Mike Lewis).
