研究人员提出Cliff方法，通过识别首次错误来改进强化学习，性能提升显著且适用性广。
Cliff是一种新的奖励塑造策略，利用现成LLM识别推理过程中的首次错误。该方法将推理分解为正确前缀和错误后缀两部分，并将此信号转换为令牌级优势。在12种不同场景的实验中，Cliff的推理性能表现优异，比在线蒸馏方法提升15%，比标准GRPO提升7%。即使使用能力适中的教师模型，Cliff也能有效提升RLVR的精细监督效果。
Cliff: Learning Process Rewards from the First Mistake
Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for large language model (LLM) post-training, but its reliance on coarse outcome rewards leads to limited guidance on intermediate reasoning processes. Existing approaches such as process reward modeling and on-policy distillation introduce additional constraints, such as reliance on a specialized reward model or assuming identical reasoning patterns between teacher and student. Nevertheless, we observe that once a reasoning process first goes wrong, evaluating the subsequent reasoning provides limited additional information, as it is already conditioned on an invalid prefix. Therefore, we propose Cliff, a reward shaping strategy that utilizes an off-the-shelf LLM as a teacher to identify the first mistake in each rollout. As a result, the rollout is naturally decomposed into two parts: a correct prefix and an incorrect suffix. Cliff then converts this signal into token-level advantages, assigning positive advantages for the correct prefix and negative feedback afterward. Experiments across 12 different scenarios demonstrate that Cliff consistently improves reasoning performance, outperforming on-policy distillation by 15% and standard GRPO by 7%, even with teachers of modest capability. Furthermore, we analyse the role of ``ground truth'' in Cliff and investigate its training dynamics. These results establish Cliff as a simple, general and effective approach for improving RLVR with richer, fine-grained supervision.