Salesforce RIVER filters defective RL environments for terminal agents
Environment quality, not environment quantity, may be the real bottleneck in agentic RL — a collection that looks large can be mostly reward noise. Teams training agents should audit verifiers before scaling data, and can adopt RIVER's filtering plus repetition penalties. River-8B's open checkpoints offer a strong 8B terminal-agent baseline against other open RL-trained models of the same size.
Salesforce AI Research audited public RL environment collections for terminal agents and found only 35.
8% of environments in the cleanest collection passed, with two others at 10.1% and 3.
3% clean.The audit identified reward errors in both directions: leaked answers and weak verifiers granting reward 1, and correct solutions getting reward 0 due to wrong reference answers.
The team's recipe, RIVER, filters defective environments and penalizes repeated commands with near-identical outputs.
River-8B averaged 19.4 across four terminal benchmarks versus 17.
7 for RL on 3.5K randomly sampled environments, and RIVER increased RL gains by 106% on Terminal-Bench-Lite and 30% on Terminal-Bench v2.
1 across 2B to 27B models using under 30% of TMax's environments.
Impressive paper from Salesforce.
It discusses the importance of good verifiers for RL environments.
Only 35.8% of the environments in the cleanest public RL collection for terminal agents passed Salesforce AI Research's audit.
More details below:
With the budget held at 3.5K environments, River-8B averaged 19.4 across four terminal benchmarks, against 17.7 for RL on 3.5K environments sampled at random from the same collection.
The audit found reward errors in both directions. Some environments give reward 1 for copying a leaked answer or passing a weak verifier without doing the task. Others give reward 0 to a correct solution because the reference answer or oracle is wrong. Two other public collections were only 10.1% and 3.3% clean.
The authors argue that RL mainly shapes behaviors, such as inspecting before acting, verifying before finishing and dropping an approach that keeps failing.
Those behaviors reuse skills the model already learned in pre-training and SFT. Their recipe, RIVER, filters defective environments and penalizes turns that repeat an earlier command with nearly the same output.
River-8B is the best of the open RL-trained 8B models they evaluated on all four benchmarks. Across models from 2B to 27B, using fewer than 30% of TMax's environments, RIVER increases RL gains by 106% on Terminal-Bench-Lite and 30% on Terminal-Bench v2.1.
Paper: https://t.co/N8NBbBMEnH
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Salesforce, RIVER, terminal agents, reinforcement learning.
Source attribution: curated from DAIR.AI (@dair_ai).
