这个新框架TRIAL能让智能体强化学习更高效，在多个任务上超过GRPO，成功率提升近19个百分点，搞RL的可以看看。
TRIAL提出统一回合对齐评分协议，为每个决策回合提取结果视图并比较普通与事后条件下的响应，以有符号对数概率差分配标记级监督。在WebShop和ALFWorld上使用不同骨干模型，TRIAL在全部八种组合中超越GRPO，并在六种组合中达到最优或并列最优。使用Qwen3-1.7B时，WebShop成功率从56.4%提升至75.2%，任务分数从78.7%提升至85.7%。消融实验显示，轨迹相对的回合分配比单纯密集事后蒸馏带来更大提升。
Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning
Recent agentic reinforcement learning methods use hindsight to complement sparse outcome rewards. However, a completed rollout can yield many such signals, leaving their appropriate allocation across turns unclear. We introduce TRIAL, a trajectory-relative hindsight distillation framework with a unified turn-aligned scoring protocol. For each decision turn, TRIAL extracts an outcome view of that decision's realized consequence and evaluates the same response under ordinary and hindsight-conditioned contexts. The signed log-probability gap determines the direction and local strength of token-level supervision, while turn-level magnitudes are normalized jointly over the realized trajectory. The resulting allocation multipliers have an eligible-token-weighted mean of one, redistributing dense supervision across turns while fixing its average multiplier. Experiments on WebShop and ALFWorld with different backbones show that TRIAL outperforms GRPO across all eight combinations of backbone, environment, and evaluation metric, while achieving the best or tied-best performance among six methods on six of them. On WebShop with Qwen3-1.7B, TRIAL improves the success rate from 56.4% to 75.2% and the task score from 78.7% to 85.7%. Controlled ablations further show that trajectory-relative turn allocation provides substantial gains beyond those of dense hindsight distillation alone.