做LVLM幻觉研究的开发者可以直接用这个无需训练的方法来提升模型可靠性，代码已开源，值得一试。
大型视觉语言模型（LVLMs）在视觉语言任务中表现出色，但仍易产生与视觉内容不一致的幻觉。研究发现，幻觉源于模型对正确视觉证据关注不足，并在生成过程中逐渐遗忘。作者提出基于层间视觉注意力差异（ILVAD）的无需训练方法，通过识别并增强对视觉证据的注意力，同时选择与视觉证据强相关的文本token进行强调。在五个最新模型上的多项基准测试中，该方法一致地缓解了幻觉，且即插即用。代码已开源。
Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy
Large Vision-Language Models (LVLMs) have shown remarkable performance on a wide range of vision-language tasks. Despite this progress, they are still prone to hallucination, generating responses that are inconsistent with visual content. In this work, we find that LVLMs tend to hallucinate when they pay insufficient attention to the correct visual evidence and gradually forget it during the generation process. We empirically find that although LVLMs overall attend insufficiently to visual evidence, they exhibit sensitivity to the correct visual evidence in specific layers, with notable inter-layer discrepancy. Motivated by this observation, we propose a novel hallucination mitigation method that enhances visual evidence based on Inter-Layer Visual Attention Discrepancy (ILVAD). Specifically, we obtain the attention weights from early generated tokens to visual tokens across layers and identify the tokens that are repeatedly activated as visual evidence, forming a saliency map. We then enhance attention to visual evidence during generation through the saliency map to reduce visual forgetting. In addition, we leverage the saliency map to obtain attention scores of generated text to visual evidence, in order to select and emphasize text tokens that are strongly grounded in visual evidence. Our method is training-free and plug-and-play. Multiple benchmark evaluations conducted on five recently released models show that our method can consistently mitigate hallucinations in different LVLMs over various architectures. Code is available at https://github.com/ytx-ML/ILVAD.