这篇论文分析了多步工具RL训练容易崩溃的原因，并实验证明交错SFT与RL能有效提升稳定性，对做智能体RL的人很有参考价值。
该论文通过实验发现，多步工具使用强化学习（RL）训练中，模型可出现灾难性崩溃，性能骤降且工具调用结构失效。根本原因是特定控制token概率突增，但基础工具使用能力并未丢失。研究者系统探索了离策略监督、提示引导、错误示例等多种监督信号，并比较了同步与交错训练方案。结果表明，将监督微调（SFT）与RL交错进行可显著提升稳定性，但在格式和内容分布外（OOD）评估中性能下降。该工作揭示了RL失败机理，并展示了多样化监督信号对鲁棒训练的价值。
Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It
Tool use enables large language models (LLMs) to perform complex tasks, and recent agentic reinforcement learning (RL) methods show promise for enhancing model capabilities. However, RL alone often leads to instability or limited gains in tool-use tasks. In our experiments, some models exhibit catastrophic collapse, where performance abruptly drops and tool-invocation structures fail. The analysis reveals that these failures stem from unexpected probability spikes in specific control tokens, disrupting structured execution, yet the underlying tool-use capability remains intact, merely obscured by specific formats. To address this, we systematically investigate a diverse set of supervisory signals, including off-policy supervision, hint-based guidance, erroneous example supervision, and others, applied under both synchronous and interleaved training schemes. We find that interleaving supervised fine-tuning (SFT) with RL substantially improves stability, but exhibits degraded performance under format and content out-of-distribution (OOD) evaluation. We also analyze the impact of learning rates and generalization across settings. These results highlight the importance of understanding RL failures and demonstrate how diverse supervisory signals can guide exploratory learning, enabling robust training of LLMs for complex, multi-step tool-use tasks. Our Code is available at https://github.com/hypasd-art/Tool-RL-Box.