BLPO：跨任务共享经验优化长程 LLM 智能体训练
BLPO: Beyond-Local Policy Optimization for Long-Horizon LLM Agents
这篇论文解决长程智能体训练中 rollout 太少、优势估不准的老问题，思路是把不同任务的经验汇到全局记忆里共享，ALFWorld 和 WebShop 上都跑出了效果，做智能体 RL 的可以看看。
arXiv 论文提出 BLPO（Beyond-Local Policy Optimization），针对长程任务中逐步组级强化学习在有限 rollout 下估计不稳定的问题，从跨任务、跨更新迭代的经验中估计任务进度优势。方法将轨迹前缀总结为任务进度状态，并按功能抽象动作，使不同任务的对应决策贡献到共享估计。在 ALFWorld 和 WebShop 上使用 1.5B 和 7B 策略做实验，在匹配训练预算下任务成功率超过所评估的基线。分析显示学习更快、轨迹更短，且无需学习 critic 或额外 rollout，计算开销很小。
BLPO: Beyond-Local Policy Optimization for Long-Horizon LLM Agents
Long-horizon tasks require large language model (LLM) agents to coordinate interdependent actions, a capability central to automating complex workflows. Stepwise group-based reinforcement learning provides finer credit than trajectory-level supervision, but reliable comparisons must account for differences in interaction history. Enforcing this consistency within the limited rollouts of each task can leave insufficient support for estimating action preferences. We propose Beyond-Local Policy Optimization (BLPO), which estimates task-progress advantages from experience shared across tasks and training updates. We summarize trajectory prefixes into task-progress states and abstract actions by their function, allowing corresponding decisions from different tasks to contribute to shared estimates. To control bias and noise from heterogeneous tasks, we accumulate context-relative returns in a global memory with balanced aggregation and reliability weighting. These global advantages complement local comparisons over normalized actions and trajectory-level credit without a learned critic or additional rollouts. Experiments on ALFWorld and WebShop with 1.5B and 7B policies show improved task success over the evaluated baselines under matched training budgets. Further analyses show faster learning and shorter trajectories with little computational overhead.
