论文：决策能力差的模型也能当好智能体的"老师"
Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents
浙大团队发现一个反直觉结论： acting 差的模型当 judge 反而能教出更强的策略。用 JEV 把 Qwen2.5-1.5B 在 ALFWorld 从 72.8% 拉到 82.9%，比 GRPO 高不少。做智能体强化学习训练的人值得把这套代码跑一下。
浙大 REAL 团队在 arXiv 发布论文《Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents》，探讨同一个决策模型在行动、评判、教学三种角色中的差异。研究在 ALFWorld 上对比了 3 个决策模型和 3 个通用 LLM，发现全部 6 个模型作为 actor 时都会被更简单的子目标带偏，跳过必须先完成的步骤；但作为 judge 看完整轨迹时，它们能给这些步骤正确评分。
机制上的关键发现是提问方式：让 judge 判断"这一步是否有帮助"而不是"任务最终是否会成功"，才能有效区分有用和无用动作。作为 teacher，JEV 方法将 Qwen2.5-1.5B 策略在 ALFWorld 上的成功率从 GRPO 基线的 72.8% 提升到 82.9%，在 WebShop 上从 56.8% 提升到 71.4%。
这对长程智能体训练有实际意义：行动能力弱的模型可以胜任奖励信号来源，因为事后评判能观察到行动时必须预判的后果。代码已在 GitHub（ZJU-REAL/decision-agents）开源。
Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents
Decision models answer a question with a typed output, such as a choice among options with a probability for each, so one model can choose an agent's actions, evaluate actions already taken, or score every action to teach a policy with reinforcement learning. Prior work has studied them mainly as actors, judging their usefulness by how well they act. We ask whether a decision model that acts poorly can still teach well. We compare three decision models and three general LLMs as actors and judges on ALFWorld, and use JEV as a teacher on ALFWorld and WebShop. As actors, all six models are pulled away from a subgoal that must come first by an easier one. As judges that see the whole trajectory, the same models credit the steps toward that subgoal, and asking whether each action helped, rather than whether the task will succeed, separates useful from useless actions. As a teacher, JEV raises a Qwen2.5-1.5B policy over GRPO from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop. These results suggest that a poor actor can still teach well, because judging an action after the episode ends reveals the consequences that acting must anticipate. Code is available at https://github.com/ZJU-REAL/decision-agents.
This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include ALFWorld, WebShop, GRPO, Qwen2.5.
Source attribution: curated from arXiv cs.AI (@Haochen Liu, Qianhao Zhu, Zixuan Wang, Dingming Li, Tongbo Chen, Peng Qiu, Yingqi Qu, Weiming Lu, Jun Xiao, Yongliang Shen).