研究揭示 Qwen-VL 与 InternVL3 的左右方向判断主要来自 ViT 位置编码
Where Do Vision-Language Models Get Left and Right? Position-Code Counterfactuals in Qwen2.5-VL, Qwen3-VL and InternVL3
做 VLM 空间理解评测或微调的人应该看看：这篇用镜像位置编码的反事实实验证明 Qwen2.5-VL、Qwen3-VL 和 InternVL3 的左右判断主要靠 ViT 位置编码，语言模型侧几乎不参与。结论直接指向关系绑定才是错误修复的正确靶点，比泛泛的评测报告有用。
一篇 arXiv 论文通过镜像 ViT 位置编码的反事实实验，定位了 Qwen2.5-VL-7B、Qwen3-VL-4B 和 InternVL3-2B 判断图像左右方向的来源。结果显示，仅镜像 ViT 宽度方向的位置编码，就能在 84.3-93.2% 的 CV-Bench 左右问答上复现翻转图像后的答案，其中 Qwen2.5-VL-3B 为 87.5%。这说明视觉侧位置编码是该能力的充分信号来源。
机制层面，在保持 ViT 编码不变的情况下，镜像语言模型侧的 token id 或 token 顺序对三个系列的左右答案影响很小，Qwen2.5-VL-7B 中该效应集中在全局注意力 ViT 块、且主要在最后几层。带定位目标的 CV-Bench 题目上，随机置换在颠倒物体顺序时比保持顺序多翻转 18.9-31.0 个百分点的答案；What'sUp 数据集上 on/under 关系比 left-right 更依赖内容与 token 顺序而非位置编码。
论文指出多数剩余错误与询问哪个物体无关，提示修复方向应在关系绑定、物体定位或参照物的使用上，而非简单调整位置编码。对从事多模态评测与可解释性的开发者有直接参考价值。
Where Do Vision-Language Models Get Left and Right? Position-Code Counterfactuals in Qwen2.5-VL, Qwen3-VL and InternVL3
A vision-language model may take left-right direction from the orientation of image content within patches, its vision transformer's (ViT) position code, or token positions and order in its language model (LLM). Ablations show the ViT code is necessary; we test sufficiency by mirroring only a position code. In Qwen2.5-VL-7B, Qwen3-VL-4B and InternVL3-2B, mirroring ViT width positions reproduces the flipped-image answer on 84.3-93.2% of CV-Bench left-right questions (Qwen2.5-VL-3B: 87.5%, partly a left fallback). With two options, this also counts disruption; a half-grid shift flips most answers too. With the ViT code intact, LLM-side id and token-order mirrors change few left-right answers in all three series (a weak id test in Qwen2.5-VL). In Qwen2.5-VL-7B the effect sits in the global-attention ViT blocks, mostly the last. On What'sUp, flipping content and token order, not the position code, keeps more on/under than left-right answers at the flipped-image answer. Post hoc analyses qualify these results. On CV-Bench items with located objects, random permutations flip 18.9-31.0 points more answers when reversing the objects' order than when keeping it. On What'sUp, in Qwen2.5-VL and Qwen3-VL, the share of the left-right changes made by random permutations that land on the mirror relation is mostly below chance. Counted as returns to the original answer, the What'sUp contrast of support with left-right relations is clear only in InternVL3-2B. Most remaining errors give the same relation whichever object is asked about, suggesting that repair may target relation binding, object localization or use of the reference object.
This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to US-China foundation model benchmarks and commercialization comparison and AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Qwen2.5-VL, Qwen3-VL, InternVL3, 多模态.
Source attribution: curated from arXiv cs.LG (@Enhao Cui, Junzhi Yang, Yaoshun Lin, Yuhan Feng, Zijian Chen).