模型一遇到难题就原地踏步？这篇论文给出了一个简单又有效的解法：给模型一点'提示'，效果直接涨3.9%，成本几乎为零。
标准RLVR在模型无法生成任何正确答案时会得到零学习信号，导致困难问题无法收敛。Off-Context GRPO通过在训练中引入解法前缀作为特权引导，使模型获得非零奖励。该方法通过重要性校正目标避免引导与原始目标的偏差，在标准数学推理基准上比vanilla GRPO平均绝对提升3.9%（相对提升13.8%），且额外成本可忽略。
Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information
Reinforcement learning with verifiable rewards (RLVR) improves reasoning in large language models. Yet, typical RLVR approaches fail on difficult problems: when a model cannot generate any correct solutions, it receives \textit{zero} learning signal. Providing privileged guidance during training, such as solution prefixes, can help overcome this learning cliff by steering the model towards {correct solutions with non-zero reward}. {We call these rollouts \textit{off-context}: they are generated from a training prompt that contains privileged guidance, while the target objective is defined by the original prompt without that guidance.} {We introduce} Off-Context GRPO (OC-GRPO), a minimally modified variant of GRPO that uses guided rollouts but applies an importance-corrected objective to steer the update back toward the original unguided objective, avoiding the mismatch that destabilizes uncorrected guided training. Empirically, our algorithm achieves a 3.9\% absolute improvement (13.8\% relative gain) over vanilla GRPO on average across standard mathematical reasoning benchmarks with negligible additional cost.