这项研究提出了Q-规划，通过离线策略Q函数与大型视觉运动BC策略的结合，实现了机器人策略的自我改进，显著提升了机器人操作的性能，值得一试。
行为克隆（BC）在机器人操作方面取得了显著进展，但其无法自我改进的局限性限制了其发展。本文提出Q-规划，通过将小型离线策略Q函数与大型视觉运动BC策略相结合，实现了价值引导的动作选择和在线自我改进。在LIBERO和双臂RoboTwin上，10次自我改进将所有基准分数提升（LIBERO-10从93%到99%，RoboTwin从83.8%到91.4%），并缩短了接近上限的套件（LIBERO-Object，LIBERO-Goal）的成功场景。在两个接触丰富的双臂真实机器人任务中，相同的循环（BC冻结，无人工干预）仅通过其自身的部署回放就纯粹提升了40%到90%（堆叠杯子）和25%到80%（插入钱包），而仅对成功回放进行SFT则停滞在55%和30%。在相同的在线预算下，Q-规划是唯一一种在Best-of-N、过滤SFT、IBRL、DSRL和DAWR中稳定地从失败中改进而不需要训练辅助演员的方法。
Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning
Behaviour Cloning (BC) has driven remarkable progress in robot manipulation, yet it is fundamentally limited by its inability to self-improve: a policy that fails cannot learn from that failure without additional human demonstrations. Reinforcement Learning fine-tuning offers a path to self-improvement but has proven difficult to scale to the multi-billion-parameter models underpinning modern robot policies. We propose Q-Planning, which equips a large visuomotor BC policy with a small off-policy Q-function. Because a Q-function estimates value rather than imitates actions, it can be trained on the same successful demonstrations as the BC policy and later absorb both successful and failed deployment rollouts, an asymmetry BC does not have. We exploit this asymmetry to enable value-guided action selection at inference (a single-step Q-weighted average over BC draws) and online self-improvement that fine-tunes only the Q-function, leaving the BC weights untouched. On LIBERO and bimanual RoboTwin, ten iterations of self-improvement lift every benchmark score we tested (LIBERO-10 93% to 99%, RoboTwin 83.8% to 91.4%) and shorten successful episodes on the near-ceiling suites (LIBERO-Object, LIBERO-Goal). On two contact-rich bimanual real-robot tasks, the same loop (BC frozen, no human intervention) improves purely from its own deployment rollouts: stack-cups 40% to 90% and insert-wallet 25% to 80% in five iterations, whereas SFT on successful rollouts alone stalls at 55% and 30%. Under an identical online budget Q-Planning is the only method, among Best-of-N, filtered SFT, IBRL, DSRL, and DAWR, that improves stably from failures without training an auxiliary actor.