27B 开源推理模型纯仿真强化学习训练真实机器人操作
Training Reasoning Models in Simulation to Natively Use Real Robots
开源模型以前玩不动机器人控制,这篇论文用纯仿真强化学习把 27B 模型练到真机上 60% 成功率,还不牺牲推理能力。做具身智能或后训练的工程师应该仔细看看 PEAR 框架的设计。
论文提出 PEAR(Post-training Engine for Agentic Robotics)训练框架,用机器人使用强化学习(RURL)在仿真环境中训练一个 27B 开源推理模型,全程无人类演示数据。模型像计算机使用智能体(CUA)一样观察环境、生成思维链推理,并直接输出下一步末端执行器位姿,无需额外封装框架。在 LIBERO 仿真基准上,训练将成功率从 1.6% 提升至 74.4%,接近 GPT-6 Astra 和 GPT-6.1 Sol 在最大推理档位下的水平。
仿真中习得的技能展现出跨域泛化能力:模型可迁移到未见过的机器人本体、真实世界以及新指令。在真实双臂机器人上,模型在 60% 的试验中完成任务,而基座模型成功率为零。与基于演示微调的视觉-语言-动作模型 π0.5 相比,该方法对任务变化的鲁棒性明显更强:在 LIBERO-Pro 上,面对熟悉场景中的新任务,π0.5 仅完成 0.75% 的回合,该模型达到 57.4%。
此外,模型在九项推理、编码与视觉基准上无可测退化,保留了通用能力。论文指出 RURL 有望成为基础模型后训练的常规组成部分,为开源权重模型追赶 GPT-6 Astra、Claude Fable 5.2 等闭源前沿模型在机器人控制上的差距提供了一条不依赖演示数据的路径。
Training Reasoning Models in Simulation to Natively Use Real Robots
Robot-use agents (RUAs) have emerged as a promising approach to general-purpose robot control. However, strong performance has so far been achieved only by frontier proprietary models such as GPT-6 Astra and Claude Fable 5.2, whereas open-weight models remain substantially less capable. We show that an open-weight 27B reasoning model can be trained to use real robots natively through reinforcement learning, entirely in simulation and without demonstrations. Like a computer-use agent (CUA), the model observes the environment, reasons in a chain of thought, and generates the next end-effector pose, without any harness. We build a Post-training Engine for Agentic Robotics (PEAR) and use it to train the model on simulated robot tasks at scale with robot-use reinforcement learning (RURL), mirroring how frontier models acquire computer use. We find that the skills acquired in simulation generalize well beyond the training tasks: they transfer to an unseen robot, to the real world, and to novel instructions. After training on LIBERO and RoboDojo in simulation, the model succeeds in 60% of our trials on a real dual-arm robot, whereas the base model succeeds in none. On LIBERO, training raises the success rate from 1.6% to 74.4%, approaching GPT-6 Astra and GPT-6.1 Sol at maximum reasoning effort. Compared with $π_{0.5}$, a vision-language-action model fine-tuned on demonstrations, our model is substantially more robust to changes in the task itself. On LIBERO-Pro, $π_{0.5}$ succeeds in only 0.75% of episodes when assigned a new task in a familiar scene whereas our model succeeds in 57.4%. Finally, the model retains its general capabilities, with no measurable degradation on nine reasoning, coding and vision benchmarks. These results highlight robot-use reinforcement learning as a promising component of foundation-model post-training.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include PEAR, LIBERO, RURL, π0.5.
Source attribution: curated from arXiv cs.LG (@Jiaming Tang, Zhuoyang Zhang, Junxian Guo, Fangzhou Zhao, Yilong Zhao, Zekai Li, Xingyang Li, Yuyang Zhao, Song Bian, Zhijian Liu, Song Han).