一种新的强化学习优化方法 Bellman Policy Optimization
Bellman Policy Optimization
推荐给做强化学习或大模型推理的研究者，BPO 是一种新的优化方法，可能对提升大模型的推理能力有帮助。
本文提出 Bellman Policy Optimization (BPO) 方法，它是一种基于策略镜像下降的批评器自由方法。该方法通过贝尔曼方程将策略优化问题转化为轨迹级别的目标函数，避免了中间状态价值估计。实验表明该方法在数学推理基准上有效。
Bellman Policy Optimization
Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models (LLMs). We introduce Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent (PMD). For autoregressive generation with terminal rewards, BPO uses the Bellman equations to reformulate PMD as a trajectory-level objective. The reformulation avoids estimating state values at intermediate states. We prove that it has the same unique optimal solution as the original PMD objective. We derive the practical BPO loss by approximating this objective. Its mismatch-correction weight is a smoothed ratio of complementary token probabilities. Experiments on mathematical reasoning benchmarks demonstrate the effectiveness of BPO.