想解决Agent长程任务奖励稀疏问题？看TRACE方法，无需额外批评器，用TD变化给每步打分，Qwen3在BrowseComp-Plus上从7分直接跳到35分。
TRACE是一种密集信用分配方法，用于代理强化学习，通过冻结参考模型获取黄金答案的log概率，转化为log比值状态值，并利用时间差分变化推导每动作奖励。在长程复杂搜索中，该方法无需冷启动监督微调或中间训练，即可显著提升基础模型工具使用能力。在封闭网络BrowseComp-Plus基准上，TRACE将Qwen3-4B从7.2提升至35.6，将Qwen3-30B-A3B从8.4提升至42.6。学习曲线显示RL训练期间更快改进和收敛。
TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
Multi-turn agents solve complex tasks through extended sequences of tool interactions before producing a final answer, making credit assignment a fundamental challenge during post-training. Outcome rewards provide reliable supervision for short-horizon reasoning, but become sparse and high-variance as trajectories grow to tens or hundreds of tool calls. They can also be misleading: a failed rollout may contain many useful actions that move the agent closer to the goal, yet outcome-only training assigns them the same negative advantage as the eventual mistake. We propose TRACE (Turn-level Reward Assignment via Credit Estimation), a dense credit-assignment method for agentic reinforcement learning. TRACE represents rollouts as state transitions at tool-call boundaries, obtains gold-answer log-probabilities from a frozen reference model, transforms them into log-ratio state values, and derives per-action rewards as Temporal-Difference changes in those values. This requires no additional critic or process-label training, and its one-step log-ratio TD component telescopes across redundant tool calls. On long-horizon complex search, TRACE substantially improves base-model tool-use ability using pure RL, without a cold-start supervised fine-tuning stage, an agentic mid-training stage, or training on live-web data. On the closed-web BrowseComp-Plus benchmark, it raises Qwen3-4B from $7.2$ to $35.6$ and Qwen3-30B-A3B from $8.4$ to $42.6$. The learned search behavior also transfers to open-web benchmarks, and the learning curves show earlier improvement and faster convergence during RL training.