多尺度策略优化 MPO 提升长程 LLM 智能体训练效果
Multiscale Policy Optimization for Long-Horizon LLM Agents
一篇关于智能体 RL 训练的新论文：MPO 把 credit 分配对齐到 turn 粒度，在 4B 和 30B-A3B 模型上稳定跑赢 RL 基线，做智能体训练的可以看看。
论文提出 Multiscale Policy Optimization（MPO），用于训练长程 LLM 智能体。核心观察是 critic 值的变化主要发生在 assistant turn 之间，因此 MPO 在 turn 粒度上锚定优化，通过 Multiscale Advantage Estimation（MAE）和 turn 级重要性采样实现更细的 credit 分配。实验在搜索、工具调用和自动化任务上，使用 4B 与 30B-A3B 两款模型验证，成绩稳定超过现有 RL 基线。
Multiscale Policy Optimization for Long-Horizon LLM Agents
Long-horizon LLM agents interact with their environments through complete assistant turns while generating each action token by token. We observe that critic values vary primarily across turns, motivating optimization anchored at turns with finer credit allocation among tokens. We propose Multiscale Policy Optimization (MPO), combining Multiscale Advantage Estimation (MAE) and turn-level importance sampling to align credit assignment and policy updates at the turn scale. Extensive experiments across search, tool-use, and automation tasks with 4B and 30B-A3B models demonstrate consistent gains over strong RL baselines.