研究区分驾驶VLA模型的指令遵循与轨迹预测
Instructions Before Actions: Prospective and Retrospective Language Supervision for Driving Vision-Language-Action Data
做自动驾驶或VLA训练的值得看这篇。它用48条路线和87个机动动作证明轨迹误差降了23-59%不等于模型真的听懂指令,还对比了Alpamayo 2 Super和Qwen-Drive对指令的响应差异。
arXiv 论文《Instructions Before Actions》研究驾驶视觉-语言-动作(VLA)模型的监督数据收集方式,将导航指令分为行动前发出的 prospective 指令与观察行动后补写的 retrospective 标签。团队构建 ADVLAT-Engine 同步语音指令、道路场景视频和车辆轨迹,在 48 条重放路线上对比 Google Maps 与两名真实导航应用,并在 87 个同时被 Google Maps 和乘客播报的机动动作上发现乘客更依赖场景的语言表达。
论文评估四个开源驾驶 VLA 在用 prospective 导航应用数据微调前后的表现:微调使乘客驾驶轨迹误差降低 23-59%,但指令影响不一致。在真实路口的反事实指令测试中,微调后的 Alpamayo 2 Super 在 15 米和 5 米处准确率分别提升 31 和 18 分,而 Qwen-Drive 用离散命令比用整句更可靠地跟随替代机动。作者还收集了四人对驾驶录像的 retrospective 标注,相关对比分析仍在进行中。
这项工作把机动预测与指令遵循区分开,为自动驾驶 VLA 的训练数据评估提供了比较 prospective 和 retrospective 监督的框架。
Instructions Before Actions: Prospective and Retrospective Language Supervision for Driving Vision-Language-Action Data
Driving vision-language-action (VLA) models can accept navigation instructions, yet predicting a driver's trajectory does not establish that a model follows the instruction. We study how driving-language supervision is collected and evaluated, distinguishing prospective instructions issued before an action from retrospective labels written after observing it. We develop ADVLAT-Engine to synchronize spoken instructions, road-scene video, and vehicle trajectories, and compare three navigation apps on 48 replayed routes. On 87 maneuvers announced by both Google Maps and a passenger, the passenger uses more scene-dependent language. We evaluate four open driving VLAs before and after fine-tuning on prospective navigation-app drives. Fine-tuning reduces passenger-drive trajectory error by 23-59%, but instruction effects remain inconsistent. At real intersections, counterfactual instructions reveal stronger responsiveness: fine-tuned Alpamayo~2 Super gains 31 and 18 accuracy points at 15 and 5~m, while Qwen-Drive follows alternative maneuvers more reliably with a discrete command than with a sentence. We additionally collected retrospective annotations from four people for the driving recordings and describe controlled comparisons of annotation content, timing, and downstream learning; those analyses are pending. Our findings separate maneuver prediction from instruction-following and establish a framework for comparing prospective and retrospective supervision.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include VLA, autonomous driving, Alpamayo 2, Qwen-Drive.
Source attribution: curated from arXiv cs.AI (@Angel Martinez-Sanchez, Ross Greer).