用信念空间规划器蒸馏训练可在不确定环境下合规行动的智能体
Deploying Physically and Epistemically Compliant Agents by Distilling Belief-Space Task and Motion Planners
机器人开发者值得看：RL 在需要主动获取信息的任务里成功率不到 5%，这套规划器蒸馏方法干到 90% 以上，还能泛化到含糊指令和更大的地图。做具身智能或任务规划的团队可以直接参考论文里的 Dual Averaging 损失设计。
一项 arXiv 论文提出将信念空间任务与运动规划器离线规划的轨迹蒸馏进策略网络，让智能体在不确定性下遵循指定的、依赖信念的规则。核心改动有两点：Dual Averaging 损失同时衡量逐步预测损失与任务变化上的平均损失，以及 ρ-Optimal Reinforcement 机制，在策略自身 rollout 接近最优时对其进行强化。论文还给出了带无噪观测模型的静态搜索问题完备性的充分条件，并证明连续信念下最大似然观测规划隐含该条件。
作者构建了名为 Scavenger 的新测试域，智能体必须主动获取信息才能完成任务，分视觉信息和自然语言指令两个版本。蒸馏后的策略在两个版本中成功率均超过 90%，而 RL 方法最多只有 5%。在泛化测试上，策略执行未见过的、故意含糊的指令成功率超过 60%，处理比训练时更大的平面图成功率超过 75%。
传统 RL 难以完美优化编码合规规则的奖励函数，信念空间规划器又需要手工编写在线规划器与信念估计器，工程复杂度高。该方法让合规行为像普通模仿学习一样可批量训练，对需要在物理与认知层面同时满足规范约束的机器人部署有直接参考价值。
Deploying Physically and Epistemically Compliant Agents by Distilling Belief-Space Task and Motion Planners
In this work, we train agents to act under uncertainty according to specified, belief-dependent rules. Deep reinforcement learning (RL) can in theory train agents to follow rules if encoded into well-designed rewards, but in practice, RL fails to perfectly optimize these rewards, leading to ineffective or noncompliant agent behavior. Belief-space planners by construction produce behavior in line with developer specification, but require hand-programming online planners and belief-state estimators, adding significant engineering complexity at scale. We combine the two: we plan offline to generate a dataset of sample trajectories and then distill the plans into a policy that acts on raw observations. We propose two changes to planner distillation that address problems specific to history-based policies: Dual Averaging, a loss that combines per-timestep predictive loss and average loss on task changes, and $ρ$-Optimal Reinforcement, a feedback mechanism that reinforces the policy's own successful rollouts when these are near-optimal. We provide a sufficient condition under which static search problems with noiseless observation models is complete, and show that maximum-likelihood-observation planning implies this condition in static search with continuous beliefs. We create a challenging new domain, Scavenger, where an agent needs to gain information in order to succeed. We evaluate our distilled policy in two versions of Scavenger, one where information is provided visually and one where it is provided as a command in natural language. In either case, the learned policy attains a success rate above 90% in Scavenger, compared with at most 5% for RL. We also find that distilled planners transfer with greater than 60% success to following unseen and intentionally vague commands and greater than 75% success to following floorplans larger than those seen in training.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and AI regulation, compliance frameworks, and sovereign compute policy. Key entities and technical topics under continuous tracking include 任务与运动规划, 强化学习, 智能体, 模仿学习.
Source attribution: curated from arXiv cs.AI (@Phillip J. K. Christoffersen, Rachel Ma, Patrick Barragán, Dylan Hadfield-Menell).