Salesforce 提出 RIVER：为终端智能体 RL 环境做质检筛选
Salesforce 这篇论文发现公开 RL 环境大多有奖励错误，还给出 RIVER 筛选方法，8B 模型跑分反超随机采样，做 RL 训练的都该看看。
Salesforce AI Research 审计发现，最干净的公开终端智能体 RL 环境集合中只有 35.8% 通过审核，另外两个公开集合的合格率仅为 10.1% 和 3.3%。审计发现奖励错误双向存在：有的环境泄露答案或验证器过弱导致错误给分，有的因参考答案错误把正确解法判为 0 分。其提出的 RIVER 方法筛选缺陷环境，并惩罚重复命令输出几乎相同的回合。在同等 3.5K 环境预算下，River-8B 在四个终端基准上平均得 19.4，高于随机采样的 17.7；使用不足 TMax 30% 的环境量，RIVER 在 Terminal-Bench-Lite 上把 RL 收益提升 106%，在 Terminal-Bench v2.1 上提升 30%。
Impressive paper from Salesforce.
It discusses the importance of good verifiers for RL environments.
Only 35.8% of the environments in the cleanest public RL collection for terminal agents passed Salesforce AI Research's audit.
More details below:
With the budget held at 3.5K environments, River-8B averaged 19.4 across four terminal benchmarks, against 17.7 for RL on 3.5K environments sampled at random from the same collection.
The audit found reward errors in both directions. Some environments give reward 1 for copying a leaked answer or passing a weak verifier without doing the task. Others give reward 0 to a correct solution because the reference answer or oracle is wrong. Two other public collections were only 10.1% and 3.3% clean.
The authors argue that RL mainly shapes behaviors, such as inspecting before acting, verifying before finishing and dropping an approach that keeps failing.
Those behaviors reuse skills the model already learned in pre-training and SFT. Their recipe, RIVER, filters defective environments and penalizes turns that repeat an earlier command with nearly the same output.
River-8B is the best of the open RL-trained 8B models they evaluated on all four benchmarks. Across models from 2B to 27B, using fewer than 30% of TMax's environments, RIVER increases RL gains by 106% on Terminal-Bench-Lite and 30% on Terminal-Bench v2.1.
Paper: https://t.co/N8NBbBMEnH
