论文 Act2Answer:评估 VLA 模型保留常识和世界知识的能力

Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models

精选理由

想知道微调后的机器人模型到底还记不记得常识?这篇论文用动作答题的方式测了7个VLA,发现简单概念还行,复杂知识掉得厉害。

AI 摘要

论文提出 Act2Answer 协议,通过让智能体在桌面场景中执行物体放置动作来选择答案,从而在动作层面评估 7 个 VLA 模型和 9 个 VLM 基线在常识与知识任务上的表现。研究发现,VLA 在简单概念上表现扎实,但在丰富语义类别上相比源 VLM 出现更大差距。实验还表明,VQA 联合训练有助于提升知识保留,而答案相关信息在 VLA 中层达到峰值,上层则衰减。

原文 · arXiv cs.LG

Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models

Embodied Vision-Language-Action (VLA) models are typically obtained by fine-tuning powerful pretrained VLMs on robotics data, yet it is unclear how much commonsense and factual knowledge they retain after adaptation. Failures on knowledge-sensitive tasks are ambiguous, conflating missing knowledge with poor generalization of low-level control. We introduce Act2Answer, a lightweight protocol that adapts VLM knowledge benchmarks to VLA evaluation by requiring agents to answer through action. Each question becomes a short tabletop episode where the agent performs a single object-placement action to select among candidate answers, yielding an action-grounded success rate with reduced control confounds. We curate a test suite of such environments across diverse commonsense and world-knowledge categories and introduce layerwise intent probing to localize answer-relevant information across the VLM backbone and action head. In a large-scale study of 7 VLA models and 9 VLM baselines, we systematically rank models across categories, finding that VLAs show solid performance on simple concepts while exhibiting larger gaps on richer semantic categories relative to their source VLMs, that VQA co-training is associated with better knowledge retention, and that answer-relevant signals peak in middle VLA layers but attenuate in upper layers. Act2Answer is available at https://tttonyalpha.github.io/act2answer/.