决策模型给长程智能体当老师，Qwen2.5-1.5B 在 ALFWorld 提升至 82.9%
这篇论文挺有意思：同一个模型当执行者会跳步，当裁判却能正确打分。研究者把决策模型当 GRPO 的老师，把 Qwen2.5-1.5B 在 ALFWorld 的成功率从 72.8% 拉到 82.9%。做智能体训练的可以看下'评估与执行分离'这个思路。
一项新研究提出，用于智能体强化学习的决策模型应只负责评估已完成的轨迹，把动作选择留给策略本身。在 ALFWorld 上，参与测试的六个模型（三个决策模型和三个通用 LLM）作为执行者时，都会先做更容易的子目标，而不是必须先做的那个；但当看到完整轨迹、被问每个动作是否有帮助时，这些模型却会给被自己跳过的步骤记分。
研究机制上，决策模型输出类型化分布，例如为每个选项给出一个概率。将其中一个决策模型用作 GRPO 的教师后，Qwen2.5-1.5B 策略在 ALFWorld 上的成功率从 72.8% 升至 82.9%，在 WebShop 上从 56.8% 升至 71.4%，其中包括它作为执行者很少能完成的任务。
这揭示了'评估'与'执行'能力的分离：模型作为评估者比作为执行者更可靠，利用这一点可以低成本提升小模型的智能体能力。论文编号为 arXiv:2611.17167。
Great paper on decision models for long-horizon agents. For agent RL, you want the decision model to evaluate completed trajectories and leave action selection to the policy. Decision models output a typed distribution, such as a probability for each option. As actors on ALFWorld, all six models tested (three decision models and three general LLMs) went for an easier subgoal before the one that had to come first. Shown the full trajectory and asked whether each action helped, the same models gave credit to the steps they skipped while acting. Used as a teacher for GRPO, one of the decision models raised a Qwen2.5-1.5B policy from 72.8% to 82.9% success on ALFWorld and from 56.8% to 71.4% on WebShop, including on tasks it rarely completes as an actor. arxiv.org/abs/2611.17167 Chat with Paper: academy.dair.ai/papers/actor-j…📊 4 ⚡
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include GRPO, Qwen2.5, ALFWorld, 智能体.
Source attribution: curated from elvis (@omarsar0). Cross-source corroboration: 2 independent sources have covered this development (including arXiv cs.AI), confirming industry-wide signal strength.
- arXiv cs.AI11-12 17:42Source
