ReflectRL 把失败样本当教材，让模型先反思再直接推理。在 9 个基准上稳定提升，还挺轻量。
ReflectRL 提出从专家模型的失败轨迹中学习推理，将失败样本视为反思对象而非模仿示范。该方法基于“反思优势”假设：对难题，反思有缺陷的轨迹比从头求解更有效。ReflectRL 先利用失败轨迹生成反思推理，再通过“反思到直接策略迁移”将能力转移。在 9 个基准、4 种 LLM 骨干和 4 种在线训练方法上，ReflectRL 稳定提升推理性能且开销极小。
ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning
On-policy training has emerged as a powerful post-training paradigm for improving the reasoning capabilities of large language models, and is often enhanced by golden trajectories from stronger expert models. However, when the expert fails on harder problems, existing trajectory-guided methods lose their main source of supervision, and these failed trajectories are typically discarded as negative samples. We argue that such failures, which we call Golden Negative Trajectories, can still provide valuable reasoning signals when treated not as demonstrations to imitate, but as flawed trajectories to reflect upon. We identify a Reflection Advantage: for hard problems, reflecting on a flawed trajectory can be easier and more effective than solving the problem directly from scratch. Motivated by this, we propose ReflectRL, a lightweight plug-and-play framework that learns from Golden Negative Trajectories during on-policy training. ReflectRL first uses these trajectories to elicit Reflective Reasoning, then applies Reflective-to-Direct Policy Transition to transfer the acquired reasoning behavior back to Direct Reasoning. Experiments across 9 benchmarks, 4 LLM backbones, and 4 on-policy training methods show that ReflectRL consistently improves reasoning performance with minimal overhead.