Meta and NYU Research Redesigns Vision Modules in Multimodal Models
The non-obvious takeaway is that the standard pipeline of pre-training a vision encoder separately may be unnecessary compute spending, and that where vision parameters sit inside the model matters more than having them at all. Teams training small multimodal models, especially edge or latency-sensitive deployments, get a concrete recipe plus faster first-token response. The 3x compute penalty for parameter sharing is a quantified warning against over-merging modalities.
Researchers from Meta and New York University published a study on the core design of vision-language models, arguing against the prevailing practice of pre-training a separate vision encoder. Their first finding: the visual encoder does not need to be trained independently in advance — training it jointly with the main model from scratch matches the image-question-answering accuracy of a separately pre-trained encoder while saving substantial compute budget. Second, the vision-specific parameters must be preserved: merging images and text into fully shared parameters sharply degrades visual capability, and recovering the loss costs roughly 3x more compute.
The paper's recommended architecture embeds the vision-only parameters inside the first eight layers of the language model. In that position, performance on text and chart recognition matches an external encoder setup, while placing the modules in middle or later layers drags results down. The design also removes the queuing step of processing images through a separate module.
The practical payoff is latency: a 1.7B-parameter model using this embedded design cut time-to-first-token from 64 ms to 37 ms on an H200 GPU. The project page with code and details is publicly available for teams building multimodal systems.
AI视觉模型的核心设计
来自Meta和纽约大学的研究，做视觉多模态模型的可以参考。核心结论：
1）视觉编码器不用提前单独训练：跟着主模型从零一起训练，最终看图问答的准确率完全不输预先单独训练的版本，而且能省下大笔算力预算。
2）必须保留专门处理图像的独立参数：如果直接删掉视觉部件、让图片和文字共用参数，看图能力会大幅掉队，需要多花3倍算力才能追平。
3）视觉专属参数放进主模型前八层最有效：把视觉独立模块直接嵌进语言模型的前八层内部，不仅识别文字和图表的效果与外挂编码器打平。放进中后层则会拖后腿。
4）嵌入式设计让AI看图出字更快：省去了图片在外挂模块排队处理的环节，1.7B规模的模型在H200显卡上的首字响应时间从64毫秒缩短到37毫秒，反应更快。
项目地址：https://t.co/V00H6W4XUF
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Meta, NYU, vision encoders, multimodal.
Source attribution: curated from Gorden Sun (@Gorden_Sun).
