做多轮RL训练智能体LLM的团队，这个静默bug可能正在破坏你的训练曲线，看完这篇分析能直接修复，省下大量调试时间。
Hugging Face CEO Clément Delangue指出，当前大多数人在用强化学习训练智能体LLM时，存在一个静默的bug：单轮RL表现完美，但加入工具调用后，损失函数会无故飙升，最终出现形状不匹配错误。根本原因在于，每次解析模型输出以检测工具调用时，重新对更新后的对话进行token化，可能导致梯度落在模型从未实际采样的序列上，从而产生无用的梯度信号。修复方法很简单：永远不要重新编码已经解码的token，将采样的token保存在一个缓冲区中，避免重新渲染。团队已发布深度分析，包括对主流开源模型家族的审计，显示大多数聊天模板已支持该修复。
Most people training agentic LLMs with RL right now have a silently broken training loop and have no...
Most people training agentic LLMs with RL right now have a silently broken training loop and have no idea. Here's the trap: single-turn RL works beautifully. Clean curves, sane rewards, everything converges. Then you add tools so the model can act mid-rollout, and things get weird. Loss spikes for no reason. Eventually a shape-mismatch error. The culprit: every time you parse the model's output to detect a tool call, then re-tokenize the updated conversation for the next turn, you're rolling the dice. Usually the round-trip gives back the same tokens. Sometimes it doesn't and your gradient lands on a sequence the model never actually sampled. No crash. Just quietly wrong math and a useless gradient signal. The fix is one rule: never re-encode tokens you've decoded. Keep the sampled tokens in one buffer, never re-render them, and both failure modes disappear. That's Token-In, Token-Out done right. Our team just published a beautiful deep-dive on exactly this, including an audit across the major open-weights model families showing most chat templates already support it. Required reading if you're doing multi-turn RL 🤗🔥 qgallouedec-tito.hf.space jM 💬 6 🔄 0 ❤️ 19 👀 988 📊 8 ⚡