Papers

论文:探究视觉基础模型的空间状态表征能力

Probing Spatial State Representations in Visual Foundation Models

Why we picked it

这篇把十个视觉模型的空间记忆能力拆成四层来测,结论挺反直觉:看得准不代表记得住,做研究的朋友可以看看评测思路。

一项研究通过空间结构、物体持久性、预测性空间更新和可操作性四个层级,对十个视觉基础模型表征进行了冻结特征探针测试。实验采用静态室内环境的流式第一人称观测,结果显示可见场景的强读出能力并不能保证物体持久性或未来特征检索的准确性。研究发现物体持久性和可操作性依赖相机位姿读出,特征预测则依赖可见场景读出。研究提出空间表征评估应关注信息能否在观测之外被维持、预测性更新并用于目标导向行动。

Original · arXiv cs.AI

Probing Spatial State Representations in Visual Foundation Models

An observer sees only part of the surrounding world, yet must keep track of what lies beyond the current view. Spatial perception therefore involves more than recognizing individual images: it calls for a persistent spatial state that is updated under self-motion and used to guide action. Do visual foundation models provide representations that support such a state? We study this question through four levels of spatial capability---spatial structure, object persistence, predictive spatial update, and actionability---using frozen-feature probes on ten representations under streaming egocentric observations of static indoor environments. The resulting capability profiles vary across tasks, network layers, and prefix lengths: strong readout of the visible scene does not ensure accurate object persistence or future-feature retrieval. Matched controls separate readout accuracy from reliance on visual history and motion. Native future prediction retains recoverable spatial structure, but a spatial reader trained on real features does not transfer to predicted features unchanged. Finally, goal-dependent candidate selection does not establish reliable continuous action utility: predicted actions respond to goal changes, yet supplying the correct goal does not reliably improve progress. Across the four levels, object persistence and actionability build on camera-pose readout, whereas feature prediction builds on visible-scene readout. Together, these findings motivate evaluating spatial representations not only by what can be decoded from them, but by whether that information is maintained beyond observation, updated predictively, and used for goal-directed action.