ECHO: 选择性记忆框架解决智能体强化学习中的历史坍塌与可溯学习

ECHO: Prune to act, trace to learn with selective turn memory in agentic RL

精选理由

这篇论文提出的ECHO框架,能让智能体在长回合任务中既保留细粒度证据,又能用强化学习追踪信用分配,效果比GRPO和SUPO都强。

AI 摘要

ECHO通过源索引重建方法,将每个环境交互回合压缩为紧凑记忆记录,并从中选择构建策略上下文。该方法在BrowseComp-Plus基准上达到43.4%保留准确率,超越GRPO的28.9%和SUPO的36.1%,且使用更少回合和轨迹量。训练后的策略在多项问答、代码生成和深度信息检索任务中实现了零样本泛化提升。

原文 · arXiv cs.LG

ECHO: Prune to act, trace to learn with selective turn memory in agentic RL

Long-horizon language agents must repeatedly interact with tools, accumulate evidence, and make decisions under bounded context windows. Existing context-management methods make such rollouts feasible by truncating distant history, folding past turns into summaries, or selecting compact memory states. However, these breakthroughs introduce two coupled limitations. First, as the number of turns grows, historical observations are progressively removed or collapsed into compressed states, making it harder for the policy to reuse fine-grained evidence. Second, once the original turns are no longer source-addressable, outcome-based RL loses an explicit path for aligning policy updates with the evidence that supported a successful final answer. To this end, we propose ECHO, a selective turn-memory framework that jointly addresses history collapse and traceable learning through source-indexed reconstruction. Specifically, ECHO compresses each completed environment turn into a compact memory record, reconstructs bounded policy contexts by selecting from these records, and reuses the selected source indices to route positive outcome credit to the evidence and selection actions that support successful answers. On BrowseComp-Plus, ECHO reaches 43.4% held-out accuracy, outperforming GRPO (28.9%) and the rolling-summary baseline SUPO (36.1%), while using fewer turns and lower trajectory volume than SUPO (Figure 1). Additionally, the trained policy improves zero-shot generalization across multi-objective QA, code generation, and deep information-seeking benchmarks on both dense and MoE backbones.