论文精选72°

FineVLA:细粒度指令对齐实现可操控的视觉-语言-动作策略

FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies

精选理由

做机器人策略学习和 VLA 模型的研究者终于有了可用的细粒度数据框架——FineVLA 不仅开源了 47K 条验证轨迹和基准,还证明了细粒度指令能显著提升操控精度,做双臂操作或仿真迁移的团队可以直接用。

AI 摘要

现有机器人数据集通常只提供粗粒度的目标级语言描述,缺乏执行细节(如活动臂、接近方向、接触区域),限制了策略的可操控性。FineVLA 提出了一个开放框架,包括数据构建工具、细粒度数据集 FineVLA-Data(47,159 条轨迹)、基准测试、专用 VLM 标注器和可操控策略。实验表明,细粒度监督不仅不牺牲目标级成功率,还能提升 1.4-8.1 个百分点,且与粗粒度指令互补,最佳混合比例(FG:Raw=1:2 至 1:1)在仿真和真实场景中均取得最高性能。细粒度监督在姿态、颜色和接近方向等关键因素上带来最大真实世界增益,建议用细粒度语言补充目标级指令。

原文 · arXiv cs.AI

FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies

Vision-Language-Action (VLA) models are increasingly expected to not only complete robot tasks, but also follow human instructions about how those tasks should be executed. However, existing robot datasets usually pair trajectories with coarse goal-level language, leaving execution-critical details such as active arm, approach direction, and contact region unspecified. This limits steerable policy learning and robotic video understanding. We introduce FineVLA, an open framework for action-aligned fine-grained VLA supervision. The framework includes: (1) a data construction tool that unifies 972,247 trajectories across 85K tasks from 10 open-source robot datasets and builds FineVLA-Data, a human-verified dataset of 47,159 fine-grained trajectories; (2) a held-out benchmark with 500 videos, 10,816 atomic facts, and 1,030 VQA questions; (3) a robotics-specialized VLM annotator for scalable fine-grained annotation; and (4) a steerable VLA policy trained with controlled mixtures of fine-grained and raw goal-level instructions. Our experiments yield three findings. First, fine-grained supervision does not sacrifice goal-level success: FG-only improves over Raw-only by +1.4 to +8.1 success-rate points across settings. Second, fine-grained and raw instructions are complementary, following a consistent inverted-U trend peaking at FG:Raw = 1:2 to 1:1. The best mixed setting reaches 86.8%/82.5% in RoboTwin simulation and 62.7/100 in real-world dual-arm manipulation (vs. 49.9 Raw-only). Third, fine-grained supervision improves steerable control: the largest real-world gains appear on pose (+23), color (+18), and approach direction (+18)--factors where goal-level instructions provide no guidance. Overall, fine-grained language should augment goal-level instructions: specifying how to execute alongside what to achieve. Project page: https://finevla.xlang.ai/