这篇论文介绍了$R^3$，一种将VLMs转化为机器人推理器的方法，通过自由形式的语言推理来指导低级策略，值得一看。
本文研究了是否可以通过训练VLMs直接在自然语言中进行推理来指导低级操作策略。引入了$R^3$，一种简单的后训练配方，将现成的VLMs转化为机器人推理器。它首先在专家生成的推理轨迹上对VLM进行中期训练，然后使用单步基于评分的强化学习从离线动作数据中改进推理器。与之前主要使用结构化轨迹作为辅助监督的机器人推理方法不同，$R^3$训练自由形式的语言推理，以产生测试时对动作的指导。在语言表格和模拟的双手购物包装两个控制测试平台上，$R^3$提高了对未见任务的探索和泛化能力，并在两个基准测试中显著优于仅基于指令的模仿学习基线。分析表明，自由形式的语言推理可以作为测试时计算机制来引导低级策略。项目页面可在https://robotic-reasoner.github.io/找到。
$R^3$: Training Robots to Reason in Natural Language via Reinforcement Learning
Reasoning in language allows foundation models to spend more test-time compute on hard problems, such as those requiring decomposition, constraint tracking, and prediction of future consequences. Whether this mechanism can improve robotic manipulation remains unclear, where long-horizon tasks require tracking partial progress, reasoning about object relations, recovering from mistakes, and steering noisy low-level policies. In this paper, we study whether VLMs can be trained to reason directly in natural language to guide low-level manipulation policies. We introduce $R^3$, a simple post-training recipe that turns off-the-shelf VLMs into robotic reasoners: it first mid-trains a VLM on expert-generated reasoning traces to initialize the desired reasoning style, then improves the reasoner with single-step rubric-based RL from offline action data. Unlike prior robotic reasoning methods that mostly use structured traces as auxiliary supervision, $R^3$ trains free-form language reasoning to produce test-time guidance for action. We instantiate $R^3$ on Language Table and simulated bimanual grocery packing, two controlled testbeds for studying robotic reasoning and long-horizon manipulation. $R^3$ improves exploration and generalization across unseen tasks and significantly outperforms instruction-only imitation learning baselines on both benchmarks. Our analyses suggest that free-form language reasoning can function as a test-time compute mechanism for steering low-level policies. Our project page is available at https://robotic-reasoner.github.io/.