操控VLM输出，像翻漫画一样准
论文发现视觉语言模型的LM骨干中存在一组称为gaze heads的注意力头，其注意力会追踪模型当前描述的图像区域。通过仅对top-100个gaze heads（少于全部9%）进行注意力掩码干预，能以83.1%的准确率引导模型描述指定的漫画面板，而随机干预无效。该干预同样适用于自然COCO图像，且机制在2B到32B参数规模及多种VLM架构中复现。该工作展示了通过机制分析实现无需重训的推理时多模态行为操控。
Gaze Heads: How VLMs Look at What They Describe
How a vision-language model internally solves the task of describing an image is far from obvious. We find that the model develops a specific mechanism for this: a small set of attention heads in its language-model backbone, which we call gaze heads, whose attention tracks the image region the model is currently describing. We find them with a simple correlation score from a few forward passes, using comic strips as a controlled testbed where narrative order is laid out spatially. These gaze heads do not just track the image tokens being described: redirecting their attention to a chosen region forces the VLM to describe that region instead. A single attention-mask intervention on the top-100 gaze heads, fewer than 9% of all heads, steers the model's answer to any chosen comic panel at 83.1% accuracy, while the same intervention on random heads fails to redirect the answer, and intervening on all heads destroys generation. The same lever also extends to continuous control: switching the gaze target mid-generation makes the model wrap up its current panel description and move to the new one within a few tokens. Beyond comics, the same intervention redirects answers to chosen regions in natural COCO images. The mechanism further recurs across model sizes from 2B to 32B parameters and across other VLM architectures, although some frozen-encoder families show no comparable head set. More broadly, this shows that targeted edits identified through mechanistic analysis can serve as practical inference-time levers for steering multimodal model behavior, without any retraining. Our code, interactive demo, and datasets are available at https://gaze.baulab.info/