多模态模型开发者常遇到生成内容偏离图像的问题，VIF用极简模块解决了视觉一致性衰减的痛点，想提升MLLM可靠性的团队可以直接试。
多模态大语言模型（MLLMs）在融合视觉与文本信息方面取得显著进展，但现有方法存在两个关键问题：视觉信息被当作普通文本token处理，削弱了其独特贡献；随着生成长度增加，模型对视觉信息的依赖逐渐减弱，导致视觉-语言对齐恶化。为此，研究者提出Vision Inference Former（VIF），一个轻量级架构模块，在推理解码阶段持续注入视觉语义，确保生成内容与视觉语义一致。在14个基准任务上的实验表明，VIF能在不同架构上稳定提升性能，且额外开销极小。代码已开源。
Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models
In recent years, multimodal large language models (MLLMs) have achieved remarkable progress, primarily attributed to effective paradigms for integrating visual and textual information. The dominant connector-based paradigm projects visual features into textual sequence, enabling unified multimodal alignment and reasoning within a generative architecture. However, our experiments reveal two key limitations: (1) Although visual information serves as the core evidential modality in MLLMs, it is treated on par with textual tokens, diminishing the unique contribution of the visual modality; (2) As generation length increases, particularly within a limited context window, the model's dependence on visual information progressively weakens, resulting in deteriorated vision-language alignment and reduced consistency between generated content and visual semantics. To address these challenges, we propose the Vision Inference Former (VIF), a lightweight architectural module that establishes a direct bridge between pure visual representations and the model's output space. Specifically, VIF continuously injects visual semantics throughout the decoding phase of the inference process, ensuring that the model remains firmly grounded in visual content during generation. We conduct experiments on 14 benchmark tasks covering general reasoning, OCR, table understanding, vision-centric evaluation, and hallucination. Experimental results show that VIF consistently improves model performance across diverse architectures while introducing minimal additional overhead. The code for this work is available at https://github.com/Dong-Xinpeng/VIF.