新书详解 RLVR 强化学习后训练的实操经验与失败案例
What Broke & Why: Practical Lessons from Reinforcement Learning Post-Training
做模型训练的工程师应该翻翻这本，五个真实训练项目把踩过的坑全摊开了：奖励被钻空子、探索停滞、评估失真这些只会在实验里遇到的问题，书里都给了排查思路，比只看 GRPO 论文实用得多。
arXiv 上发布的新书（编号 2611.10844）系统讲解基于可验证奖励的强化学习后训练（RLVR），覆盖 GRPO 算法与训练循环的逐步拆解。书中总结了训练失败的四大类原因：模型拿不到有效奖励、学会钻奖励系统的空子、停止探索、以及评估指标本身失真导致表面提升。
正文围绕五个完整训练项目展开，分别涉及数学推理、搜索 Agent、训练稳定性、MoE 混合专家模型和软件工程任务，每个项目都记录了遇到的问题、调整方案和结论。后半部分作为实用参考手册，讲解奖励设计、评估方法、工具使用型 Agent 训练、常见故障调试、rollout 引擎与硬件配置。
RLVR 目前是提升开源模型推理能力的主流路线，但公开资料多为成功案例，失败细节很少被记录。这本书面向做开源模型训练的研究者和工程师，价值在于把'为什么坏了'写清楚，帮助判断下一步该改哪里。
What Broke & Why: Practical Lessons from Reinforcement Learning Post-Training
Reinforcement learning with verifiable rewards (RLVR) has become an important way to improve how language models reason, solve problems, and use tools. But getting it to work in practice is not always straightforward. Training can fail because a model receives no useful reward, learns to exploit the reward system, stops exploring, or appears to improve when the evaluation itself is misleading. In this book, we give a practical introduction to reinforcement learning post-training for language models, focusing on what works, what fails, and why. We begin with the foundations of post-training, explain GRPO and the training loop step by step, and then follow five training programs covering math reasoning, search agents, training stability, mixture-of-experts models, and software engineering tasks. Each program shows the problems encountered, the changes made, and the lessons learned. The book also examines what reinforcement learning actually changes inside a model. The remaining chapters provide a practical reference for reward design, evaluation, training tool-using agents, debugging common failures, reproducibility, rollout engines, and hardware. Written for researchers and practitioners working with open models, the book aims to help readers understand how to train models, recognize when something goes wrong, and decide what to try next.
This report is categorized under AITOP's "How-to" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include GRPO, RLVR, 强化学习, 后训练.
Source attribution: curated from arXiv cs.AI (@Luv Verma).
