想提升LLM强化学习效果？3PO用参数采样替代温度调节，在OLMo和Qwen上比GRPO更稳更好，成本还一样。
arXiv新论文提出参数空间探索方法3PO（Perturbed Parameter Policy Optimization），通过从后验分布采样不同策略生成rollout，补充传统动作空间探索的不足。在OLMo-3-1025-7B和Qwen2.5-Math-7B上，3PO在数学推理和代码生成任务中平均性能优于标准GRPO，且FLOPs成本几乎相同。多参数采样还减少了训练中零优势组和错误rollout的数量。
Parameter Exploration for RLVR via Variational Learning
Exploration has been a focus of reinforcement learning research for a long time. Recently, there has been growing evidence that it is also an important ingredient in LLM reinforcement learning recipes that can significantly impact downstream performance. Many existing methods control exploration in the action-space, for example, using temperature scaling. However, these methods cannot reorder tokens but only influence the variance in the output distribution. This limits exploration and can lead to divergence or stalled training. Here, we investigate parameter-space exploration, where rollouts are generated by sampling different policies from a posterior that may each explore different rollouts. Sampling less or more diverse policies is then a complementary control lever over exploration. We introduce a family of methods called Perturbed Parameter Policy Optimization (3PO) which use different sampling strategies and different rollout grouping for reward estimation. Experiments on OLMo-3-1025-7B and Qwen2.5-Math-7B across mathematical reasoning and code generation tasks show that these approaches consistently improve average downstream performance over standard GRPO at a near-identical FLOPs cost. Moreover, using multiple parameter samples consistently produces fewer zero-advantage groups and malformed or incorrect rollouts during training than GRPO and action-space baselines. Overall, our work presents evidence that parameter-space exploration can improve reinforcement learning for LLMs.