VPO解决了LLM在推理时搜索中缺乏多样性的痛点，做强化学习后训练或推理时搜索的团队值得关注，它直接替换GRPO就能提升搜索效果。
当前大语言模型的后训练通常优化单一标量奖励，导致输出分布低熵，难以在推理时搜索（如AlphaEvolve）中展现多样性。本文提出向量策略优化（VPO），一种显式训练模型适应多样化下游奖励函数并产生多样化解决方案的强化学习算法。VPO利用实践中奖励常为向量形式（如代码生成的逐测试用例正确性、多种用户画像或奖励模型），作为GRPO优势估计器的即插即用替代，训练模型输出一组解，每个解专门针对向量奖励空间中的不同权衡。在四个任务上，VPO在测试时搜索（如pass@k和best@k）中匹配或超越最强标量RL基线，且差距随搜索预算增大而扩大。对于进化搜索，VPO模型能解决GRPO模型完全无法解决的问题。
Vector Policy Optimization: Training for Diversity Improves Test-Time Search
Language models must now generalize out of the box to novel environments and work inside inference-scaling search procedures, such as AlphaEvolve, that select rollouts with a variety of task-specific reward functions. Unfortunately, the standard paradigm of LLM post-training optimizes a pre-specified scalar reward, often leading current LLMs to produce low-entropy response distributions and thus to struggle at displaying the diversity that inference-time search will require. We propose Vector Policy Optimization (VPO), an RL algorithm that explicitly trains policies to anticipate diverse downstream reward functions and to produce diverse solutions. VPO exploits that rewards are often vector-valued in practice, like per-test-case correctness in code generation or, say, multiple different user personas or reward models. VPO is essentially a drop-in replacement for the GRPO advantage estimator, but it trains the LLM to output a set of solutions where individual solutions specialize to different trade-offs in the vector reward space. Across four tasks, VPO matches or beats the strongest scalar RL baselines on test-time search (e.g. pass@k and best@k), with the gap widening as the search budget grows. For evolutionary search, VPO models unlock problems that GRPO models cannot solve at all. As test-time search becomes more standardized, optimizing for diversity may need to become the default post-training objective.