机器人研究者终于有了一个专门测试认知推理和意外应对的基准——RoboWits 揭示了 VLA 模型在变异任务上的脆弱性，做机器人操作和具身智能的团队值得关注这个评估框架。
RoboWits 是一个双手机器人基准测试，旨在系统评估机器人在意外条件下的认知推理、创造性工具使用和鲁棒性。研究团队提出了一个多智能体协作框架，自动生成包含几何、材料和装配推理的 30 个种子任务和 208 个变异任务。测试发现，预训练的视觉-语言-动作模型（VLA）在种子任务上表现尚可，但在变异任务上表现脆弱，无法应对需要推理和策略适应的操作场景。这表明当前机器人策略在创造性问题解决方面存在显著差距。
RoboWits: Unexpected Challenges for Robotic Creative Problem Solving
The ability to reason, adapt, and creatively solve problems under unexpected challenges is essential for robots operating in real-world environments. However, current robotic benchmarks primarily emphasize skill-level execution and provide limited insight into such cognitive reasoning capabilities. We introduce RoboWits, a bi-manual robotic benchmark designed to systematically evaluate cognitive reasoning, creative tool use, and robustness to unexpected conditions. To enable scalable construction of high-quality reasoning-centric unexpected scenarios, we propose an automated task generation pipeline formulated as a multi-agent cooperative framework, comprising agents for seed task generation and verification, metric generation, scene generation, and task mutation. Using the pipeline, we curated 30 diverse seed tasks and 208 tasks with mutations and graded difficulty across geometry, material, and assembly-based reasoning. We benchmark popular robot policies, pre-trained VLAs, and oracle-state planners. Our results reveal a significant performance gap: while pre-trained VLAs exhibit preliminary success on seed tasks after single-task fine-tuning, they struggle to perform on mutated tasks, implying their brittleness in manipulation tasks requiring reasoning, strategy adaptation, and robustness to deceptive or constrained environments. Project page is available at https://umass-embodied-agi.github.io/RoboWits.