VideoLVR:用潜空间视觉证据改进视频推理
VideoLVR: Latent Visual Reasoning over Grounded Video Evidence
一篇讲视频推理的新论文,思路是让模型在思考时能回看关键画面片段,不用塞进一堆帧,做视频理解方向的可以看看。
VideoLVR 在视频推理的中间步骤中引入潜空间视觉证据,从相关视频时刻的局部区域构建视觉支持,缓解语言推理链带来的视觉遗忘问题。它提出视觉 token 预测目标,将视觉证据重建和文本生成统一到同一个 token 预测框架。进一步扩展到强化学习时,模型把采样的视觉 token 与文本 token 一同作为动作,用最终答案奖励联合优化策略。该方法无需大幅扩展多模态上下文即可在多个视频基准上提升效果,对需要细粒度视觉证据的任务尤其明显。
VideoLVR: Latent Visual Reasoning over Grounded Video Evidence
Chain-of-thought reasoning has substantially advanced video understanding in multimodal large language models (MLLMs). However, language-based reasoning traces do not provide a reliable mechanism for models to revisit relevant visual evidence, leading to visual forgetting and impairing their ability to reason over video content. Existing approaches maintain visual grounding during reasoning in two main ways: they either refer to evidence using textual coordinates, which specify symbolic locations rather than the visual content itself, or reinsert cropped regions or frames into the multimodal context, preserving visual details at the cost of substantial context expansion. To address these limitations, we introduce VideoLVR, which constructs latent visual evidence from local regions at relevant video moments, thereby providing visual support for intermediate reasoning steps. To supervise latent visual evidence generation, we introduce a visual token prediction objective that formulates visual token reconstruction as prediction over a vocabulary derived from the input video, thereby unifying visual evidence reconstruction and text generation within the same token prediction framework. This shared prediction framework enables MLLMs to reason over interleaved text tokens and latent visual states while integrating evidence from different moments. We further extend VideoLVR to reinforcement learning by treating sampled visual tokens as actions alongside text tokens and jointly optimizing the policy using final-answer rewards. By efficiently revisiting task-relevant visual information during reasoning, VideoLVR improves video understanding without substantially expanding the multimodal context. Experiments on multiple video benchmarks demonstrate its effectiveness, especially on tasks requiring fine-grained visual evidence.