做连续控制RL的理论研究者会感兴趣——这篇论文用Bellman结构替代凸性假设，为WPG的全局收敛提供了首个完整证明，建议做策略梯度理论的团队仔细读。
该论文首次为Wasserstein策略梯度（WPG）方法在熵正则化强化学习中的全局收敛性提供了严格理论证明。WPG利用动作分布的最优传输几何，通过软Q函数的动作梯度与Langevin扩散更新策略，但标准Langevin分析因RL目标通过Bellman递归依赖策略而失效。研究者通过Bellman残差的KL表示、Bellman收缩与Bellman预解恒等式，建立了分布Polyak-Łojasiewicz条件，并利用对数Sobolev不等式控制离散化误差，最终证明WPG以几何速率收敛至全局最优（存在离散化偏差）。该工作揭示了熵正则化RL虽非传统凸优化，但Bellman递归诱导了有利的PL几何结构。
Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learning
Wasserstein policy gradient (WPG) is a policy optimization method for reinforcement learning (RL) that exploits the optimal-transport geometry of action distributions. For the entropy-regularized RL objective, WPG evolves each state-conditional policy by transporting it along the action gradient of the soft Q-function together with a Langevin-type diffusion. Despite its appeal for continuous-control problems, its global convergence properties remain poorly understood. Standard Langevin analyses do not directly apply, because the RL objective depends on the policy through the Bellman recursion rather than through a static convex functional, and the Langevin drift is determined by the soft Q-function, whose regularity must be controlled along the policy iterates. In this paper, we develop a global convergence theory for WPG by exploiting the Bellman structure of entropy-regularized RL. We show that the role usually played by convexity can be replaced by a Bellman-based argument: the soft Bellman residual admits a statewise KL representation with respect to a Gibbs policy; Bellman contraction relates this residual to the global optimality gap; and a Bellman resolvent identity connects value improvement to relative Fisher information. Combined with a uniform log-Sobolev inequality (LSI) for the evolving Gibbs family, these ingredients yield a distributional Polyak--Łojasiewicz condition. We further establish the regularity and uniform bounds needed to control the discretization error, thereby obtaining geometric contraction up to a discretization bias. Conceptually, our analysis shows that although entropy-regularized RL is not convex in the usual flat sense, the Bellman recursion induces a favorable Polyak--Lojasiewicz-type (PL) geometry that supports global convergence of WPG.