论文精选

Closed-Loop Trace Distillation:让VLM正确读取机器人探索操作轨迹

When Video Misreads: Closed-Loop Distillation of Reading Heuristics for Exploratory Manipulation Trace QA

精选理由

机器人操作中失败尝试常被忽略,但本文证明它们才是关键线索——做具身智能或机器人规划的团队,可以用蒸馏出的单行提示直接提升VLM的轨迹理解能力,值得在仿真和真实场景中试试。

AI 摘要

机器人探索操作中,一次看似失败的尝试(如拉锁住的抽屉)往往揭示了完成任务的关键前提条件。本文形式化定义了探索操作轨迹问答(EMT-QA)任务:给定同步视频和本体感知数据,预测最小成功动作链。现有VLM和具身多模态大模型无法可靠地从原始数据中恢复该链条。作者提出闭环轨迹蒸馏(Closed-Loop Trace Distillation)方法,通过每个任务的编码代理检查标注轨迹,蒸馏出一行自然语言提示(DRH)。在三个模拟器和两个真实机器人任务上,DRH将链条预测准确率提升0.38-0.47,且DRH本身可作为一次性程序化分类器的唯一规范。

原文 · arXiv cs.AI

When Video Misreads: Closed-Loop Distillation of Reading Heuristics for Exploratory Manipulation Trace QA

Exploratory manipulation often turns an apparent failed attempt into the key evidence for what to do next. For example, a robot pulls a locked cabinet drawer, fails, and only succeeds after opening the lock. The failed pull reveals a latent precondition (the drawer is locked) that determines the minimal-success action chain (the fewest actions that complete the task), here [lock-open, drawer-pull]. Correctly reading this trace is therefore the prerequisite for recovering that chain. We formalize this setting as Exploratory Manipulation Trace QA (EMT-QA): given synchronized video and proprioception from an exploratory trace, predict the minimal-success action chain under the latent precondition revealed by the probe. However, even state-of-the-art VLMs and embodied multimodal LLMs misread this evidence: they do not reliably recover the chain from raw video, raw proprioception, or their combination. We introduce Closed-Loop Trace Distillation, a pipeline that uses a per-task coding agent to inspect labeled training traces and distill a one-line natural-language prompt over the trace, which we call the Distilled Reading Heuristic (DRH). At inference, no agent is invoked and no model weights are updated; a frozen VLM receives the raw trace plus the DRH as a prompt entry. Across three simulator and two real-robot tasks, the DRH improves chain accuracy by +0.38 to +0.47 over the best raw-modality baseline. The same DRH also serves as the sole specification for one-shot programmatic classifiers that match the prompted VLM.