论文精选73°

TASPO解决智能体政策优化中的监督-信用差距

Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization

精选理由

TASPO解决了智能体政策优化中的监督-信用差距问题,让特权信息只重新分配行动间的信用,不改变整体更新方向。

AI 摘要

TASPO将特权监督转化为结果导向的行动信用,在三个智能体基准测试中比GRPO提高10.6%。该方法从验证的成功经验中构建决策适用的特权信息,在可执行行动级别聚合PI引起的似然变化。TASPO将相对行动支持转换为原始轨迹优势上的正、有界、均值保留权重,验证结果确定更新方向和平均规模。

原文 · arXiv cs.AI

Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization

Outcome-based reinforcement learning provides verified feedback for language-model agents, but assigns trajectory-level advantage uniformly to all decisions, yielding coarse credit over long-horizon interactions. On-policy self-distillation offers finer supervision by re-evaluating sampled behavior with privileged information (PI) available only during training. However, fine-grained supervision is not necessarily fine-grained credit: PI-induced likelihood changes describe how additional information alters policy preference, but do not directly determine how an executable action should inherit the verified task outcome. This creates a supervision-credit gap. Privileged signals may be irrelevant to the current interaction state, operate at a token granularity misaligned with executable decisions, and lack the outcome semantics required for reinforcement. We introduce TASPO, which converts privileged supervision into outcome-grounded action credit. TASPO constructs decision-applicable PI from verified successful experience, aggregates PI-induced likelihood shifts at the executable-action level, and converts relative action support into positive, bounded, mean-preserving weights on the original trajectory advantage. Thus, the verified outcome determines the update direction and average scale, while PI only redistributes credit across actions. Across three agentic benchmarks, TASPO improves over GRPO by 10.6\% and generalizes better to unseen tasks. Further analysis indicates that TASPO reduces supervision mismatch and that action-level assignment stabilizes the policy optimization process. These findings offer the community another interesting perspective.