做智能体强化学习或 RLVR 的团队，TRACE 解决了多轮 rollout 中奖励信号稀疏的痛点，直接用树状分配提升采样效率，值得在自家 agent 训练流程中试试。
TRACE 提出了一种针对多轮智能体强化学习（RLVR）的 rollout 预算分配框架，解决了因奖励对比不足导致的策略优化效率低问题。传统方法仅在 prompt 层面分配资源，忽略了同一 rollout 中不同回合（turn）前缀的信息量差异。TRACE 将每个 ReAct 风格的思考-行动-观察回合建模为语义节点，形成树状结构，并动态分配预算到最可能产生混合奖励的 prompt 根节点和中间前缀。实验表明，在相同采样成本下，TRACE 在 Multi-Hop QA 等基准上将 Qwen3-14B 的平均准确率提升了 2.8 个百分点。该框架通过增强奖励对比，显著提升了多轮智能体任务的训练效率。
TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning
Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models. However, rollout-intensive policy optimization is often limited by insufficient reward contrast, arising when overly simple or complex prompts generate low-variance feedback and when outcome-only rewards assign the same terminal assessment to every decision in a multi-turn rollout. Past efforts have focused on allocating available rollout resources to promising prompts, yet they only leverage sample informativeness at the prompt level and neglect variation in prefix-level informativeness across turns within the same rollout. This work targets multi-turn agentic RL by modeling each ReAct-style thought-action-observation turn as a semantically distinct node, allowing budget allocation to extend from prompt roots to turn-level prefixes with further continuations, which naturally forms tree-structured rollouts. We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework that enhances reward contrast within a fixed sampling budget. Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards. A shared generalizable predictor estimates conditional success probability at these anchors from prefix histories to guide this allocation. The resulting adaptive tree structure enriches outcome-only feedback and amplifies the policy-update signal. Empirically, TRACE achieves competitive performance and efficiency gains on typical agentic benchmarks, e.g., improving Qwen3-14B Multi-Hop QA average accuracy by 2.8 points over competitive baselines at equal sampling cost.