做开放域文本生成（如角色扮演、创意写作）的团队，终于有了一个兼顾对齐质量和输出多样性的RL方法，不用再担心模型输出千篇一律，值得点开看实现细节。
现有强化学习方法在可验证任务中表现优异，但在开放生成任务中面临奖励模型训练成本高、输出多样性差的问题。研究者提出PPR-GDE方法，无需标量奖励，通过成对偏好奖励保留主观评价的比较结构，并引入群体级多样性奖励显式鼓励语义分散。该方法在角色扮演任务上实现了比强基线更好的对齐质量和表达多样性。实验表明，成对偏好对主观偏好对齐至关重要，而多样性指标对实现更广的语义覆盖不可或缺。
Pairwise Preference Reward and Group-Based Diversity Enhancement for Superior Open-Ended Generation
Current reinforcement learning(RL) methods are broadly applicable and powerful in verifiable settings where scalar rewards can be provided. However, in open-ended generation tasks, verifying the correctness of responses remains challenging, and training reward models incurs substantial computational and annotation costs. Moreover, reinforcement learning (RLVR) often leads to diversity collapse and produces stereotypical or rigid outputs, outcomes that are particularly undesirable in open-domain scenarios. We propose Pairwise Preference Reward and Group-based Diversity Enhancement (PPR-GDE), a RL method that is more suitable for open-ended generation. PPR-GDE does not require scalar rewards and incorporates group-level diversity into the reward signal, it preserves the comparative structure of subjective evaluation through a pairwise preference reward, mitigates judge position bias via repeated comparisons with swapped response order, and introduces a group-based diversity reward that explicitly encourages semantic dispersion within a response group, all of these reward signals are integrated into a unified group-relative policy optimization objective. We instantiate PPR-GDE on role-playing task, experiments show that PPR-GDE achieves a better alignment quality as well as expressive diversity than strong RL baselines. Further analysis shows that pairwise preference is critical for preference alignment in subjective perspective, while the diversity metric plays an essential role in achieving superior expressive diversity and broader semantic coverage.