研究揭示 VLM 描述图像时如何规划物体列举顺序
How VLMs Plan What's Next
这篇论文把 Qwen3-VL 生成描述时的内部机制拆开了:怎么定位物体、怎么记住写到哪里、怎么决定下一个看哪。用因果补丁验证,2B 到 32B 都成立。做可解释性或多步规划研究的读者值得拿来当起点。
一篇 arXiv 论文通过因果补丁实验刻画了 Qwen3-VL 模型在描述图像时的规划机制,回答了视觉语言模型如何决定"下一个看哪里"的问题。该机制包含四个环节:binding heads 将相关文本 token 与图像匹配并解析为指向图像区域的指针;同样的 binding heads 把已写出的答案绑定到来源区域,形成类似光标的进度信号,标记列表已写到图像的哪个位置。
接下来,select 步骤在光标之后选择最近的未写区域或决定停止,gaze heads 则执行选定的指针并返回该区域的内容概念,再由语言 heads 和后期 MLP 转化为回答中的下一个词。验证实验在 Qwen3-VL 上基于合成视觉检索任务进行,并通过与替代假设对比来排除其他解释。
结果显示该机制在多种输入格式、自然照片和不同语言下均成立,binding、relay 和 gaze heads 在 2B 到 32B 的所有模型规模上都存在。项目页面位于 plan.baulab.info,这项工作为逐层、逐 token 地追踪 VLM 如何把问题和图像转化为答案序列提供了方法。
How VLMs Plan What's Next
When a vision-language model describes an image, it names one object at a time and it rarely repeats itself, yet how the model plans this order is far from obvious. We characterize the mechanism a VLM uses to decide where to look before it looks. The mechanism has four main parts. First, binding heads match the relevant text tokens against the image and resolve them into pointers to the matched image region. Second, the same binding heads bind each written answer to the region it came from, and the model turns it into a progress signal: a cursor, the position in the image up to which the list has been written. Third, a select step chooses the nearest unwritten region after the cursor, or stops. Finally, gaze heads execute the chosen pointer and return a concept of what is drawn there, which language heads and the late MLPs turn into a word in the language of the question. We validate each part of this mechanism through causal patching in Qwen3-VL models on a synthetic visual retrieval task, and test our mechanisms against alternative hypotheses. We further show that the mechanism holds across input formats, natural photographs, and languages, and that its binding, relay, and gaze heads appear at every model size from 2B to 32B. This work allows us to investigate how a VLM turns a question and an image into a sequence of answers, layer by layer and token by token. More broadly, it opens a way to study how models plan over complex multi-step tasks with long rollouts. Our project is available at https://plan.baulab.info/
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Qwen3-VL, VLM, interpretability, vision-language models.
Source attribution: curated from arXiv cs.LG (@Rohit Gandikota, Arnab Sen Sharma, Adam Belfki, David Bau).