Critical-State RL Improves Multi-Turn Tool Use in Salesforce Paper
The insight is that most reward variation in multi-turn RL comes from downstream noise, so credit assignment should target the pivotal call rather than the full trajectory. Researchers and engineers building tool-using agents can apply nested sampling and contextual-bandit updates to concentrate training on decision-critical states. The BFCL v4 result offers a concrete benchmark for comparing against trajectory-level reward approaches.
Salesforce AI Research published a paper on reinforcement learning for multi-turn tool use.
The method, Critical-State RL, trains only the call where the action changes the outcome instead of spreading reward across the whole trajectory.
It uses nested sampling to separate reward variation caused by the current action from downstream noise, then applies contextual-bandit updates to the selected call.
On BFCL v4 missing-function tasks, training the selected turn adds about 14 points, while training another candidate turn leaves accuracy flat or lower.
Great paper from Salesforce AI Research on RL for multi-turn tool use.
The finding is that you want to train the one call where the action changes the outcome, instead of spreading reward across the whole trajectory.
(bookmark it)
When reward depends on later turns, much of its variation comes from what happens downstream.
Critical-State RL uses nested sampling to separate the reward variation caused by the current action from that noise, then trains only the selected call with contextual-bandit updates.
On BFCL v4 missing-function tasks, training the selected turn adds about 14 points. Training the other candidate turn leaves accuracy flat or lower.
Paper: https://t.co/vWtoTEVq1a
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Salesforce AI Research, Critical-State RL, BFCL, tool use.
Source attribution: curated from DAIR.AI (@dair_ai).
