这篇论文搞了个VLM-IE3D，让普通视觉语言模型看RGB视频就能学会3D空间感，不用额外3D设备，而且好几个3D任务都做得更好。
现有视觉语言模型（VLM）在处理3D任务时因缺乏空间理解而表现受限。研究者提出VLM-IE3D框架，通过从RGB视频中学习隐式几何令牌（IGTs）和显式几何令牌（EGTs），增强3D空间感知能力。该框架采用3D感知适配器融合两种几何表示与2D视觉信息，无需额外3D输入。在3D视频检测、3D视觉定位、3D密集描述和空间推理等任务上，VLM-IE3D均取得一致性优越性能。代码和模型已开源。
3D-Aware VLMs with Implicit and Explicit Geometries
Despite rapid progress, most existing vision-language models (VLMs) built from 2D visual inputs often struggle when handling various 3D tasks that require fine-grained spatial understanding and reasoning. To bridge this gap, we present VLM-IE3D, a unified framework that enhances the 3D spatial awareness of VLMs by equipping them with both implicit and explicit 3D geometries learned from RGB videos. Our VLM-IE3D introduces Implicit Geometry Tokens (IGTs) that capture high-level geometric priors from input videos, as well as complementary Explicit Geometry Tokens (EGTs) that encode detailed geometric structures from reconstructed 3D attributes. On top of that, VLM-IE3D comes with a 3D-aware adapter that effectively fuses the two types of geometric representations with 2D visual cues. This RGB-only design injects strong 3D inductive biases for fine-grained spatial understanding and reasoning without requiring any additional 3D inputs. Extensive experiments show that VLM-IE3D achieves superior performance consistently across various 3D tasks including 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning. Code and models are available at https://github.com/Vegetebird/VLM-IE3D.