这篇论文提出 RRC，能把生成式奖励模型的排序结果转成强化学习能用的信号，在对话和推理任务上比现有方法更稳定地提升训练效果。
生成式奖励模型在响应排序上表现出色，但其比较性输出与强化学习要求的标量奖励不匹配。为此论文提出 RRC（Ranking-based Reward Construction），通过相对偏好排序构造奖励信号。RRC 包含自竞争排序和锚定引导排序两种互补策略，分别利用采样响应间的比较和小规模参考响应实现可扩展构造。实验在开放对话和推理基准上验证，RRC 相比现有方法持续提升强化学习训练效果。
RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction
Recent advances in reward modeling show a paradigm shift from discriminative reward models to generative reward models. However, despite their strong capabilities in response ranking, generative reward models have not realized their potential in reinforcement learning (RL). Our analysis reveals that this limitation arises from a mismatch between the comparative nature of generative reward modeling and the scalar scoring paradigm adopted by existing RL algorithms. To bridge this gap, we propose a Ranking-based Reward Construction (RRC) approach, which enables generative reward models to provide more effective RL learning signals by deriving rewards from relative preference rankings. RRC introduces two complementary strategies: self-competitive ranking, which exploits comparisons among sampled responses, and anchor-guided ranking, which enables scalable ranking-based reward construction with a small set of reference responses. Experiments across open-ended chat and reasoning benchmarks demonstrate that RRC substantially improves RL training with generative reward models, achieving consistent gains over existing reward construction approaches. Our code can be found at https://github.com/wangclnlp/RRC.