L2RR：用目标端反馈优化 GRPO rollout 采样分布
Dataset Mismatch Matters in Group Relative Policy Optimization for Reinforcement Learning from Verifiable Rewards
做 RLVR 训练的朋友可以看看：目标集提示不能外传时，用少量聚合反馈就能重加权 GRPO 的采样分布，数学基准上验证有效。
论文提出 L2RR（learning-to-reweight rollout），针对 RLVR 训练中源数据集与目标测试集分布不一致的问题。方法通过双层优化学习源提示的离散采样分布，最大化 held-out 目标集准确率，同时惩罚低信息增量的 rollout。实验覆盖 6 个数学推理基准和 3 个模型规模，结果显示目标域性能和 rollout 效率均提升，且无需向训练方共享目标提示或 rollout 轨迹。
Dataset Mismatch Matters in Group Relative Policy Optimization for Reinforcement Learning from Verifiable Rewards
Recently, reinforcement learning from verifiable rewards (RLVR) is practically very important, where the leading approach is group relative policy optimization (GRPO) that typically faces a trade-off between rollout efficiency and downstream performance. To improve efficiency while preserving performance, existing selective rollout methods mainly exploit only source-side training dynamics to optimize the discrete distribution over source prompts. However, in practice, the source training dataset is rarely identically distributed to the target test dataset. When such a mismatch exists, due to legal or privacy constraints, it is usually not acceptable to share target prompts themselves with the RLVR trainer. Nevertheless, it is still possible for the target data owner to provide limited aggregated feedback from a small held-out target set without revealing target prompts or rollout traces. In this paper, we propose learning-to-reweight rollout (L2RR) that explicitly incorporates target-side performance feedback into GRPO. Specifically, L2RR learns a discrete sampling distribution over source prompts by optimizing a bi-level objective that maximizes accuracy on a held-out target dataset, while simultaneously penalizing rollouts that provide little incremental learning information. Empirical studies on six math reasoning benchmarks and three model scales show that L2RR improves target-domain performance and increases rollout efficiency.
