这篇论文提出MMCS，用50K样本就干翻别人600K的效果，做多模态预训练的可以看看这个新思路。
MMCS是一种新的多模态预训练方法，通过将文本实体替换为对应视觉对象，提供显式的对象级监督。该方法仅用50K样本就能匹配或超越在600K图像-文本对上训练的模型，数据效率提升12倍。MMCS在773K样本的预训练数据集上验证了有效性，并持续提升不同规模模型的视觉定位和感知能力。该研究针对现有MLLMs中图像级对齐的指称歧义问题，提出了更精确的局部视觉-语言对齐方案。
MultiModal Code-Switching: Interleaving Visual Objects into Language for Explicit Object-Level Alignment
Existing Multimodal Large Language Models (MLLMs) predominantly rely on image-text pairs for modality alignment pretraining, mapping global image representations to long textual descriptions. However, this image-level alignment suffers from referential ambiguity: models struggle to infer the correspondences between multiple visual objects and textual entities from the global representation, leading to data inefficiency and suboptimal semantic grounding. To address this, we propose MultiModal Code-Switching (MMCS), a novel pretraining paradigm that provides explicit object-level supervision. Inspired by the linguistic phenomenon of code-switching, MMCS interleaves vision and language by replacing textual entities with their corresponding visual objects, enforcing local vision-language grounding. We further develop a scalable data synthesis pipeline to generate a pretraining dataset of 773K samples with accurate object-entity correspondences. Experiments show that MMCS is highly data-efficient: with only 50K samples, it matches or surpasses models trained on 600K image-text pairs. Furthermore, MMCS consistently improves visual grounding and perception capabilities across varying model scales.