这篇论文用Lyapunov函数严格证明了策略梯度在扩散环境老虎机里的收敛性，遗憾界做到O(log T)，搞理论的人可以看看。
该论文在Wang等(2020)和Jia与Zhou(2022b)的连续时间强化学习框架下，研究扩散环境中多臂老虎机问题的策略梯度更新。使用logit参数化随机策略时，作者证明在任意常数学习率下算法几乎必然收敛到最优臂。此外，当常数学习率低于一个时不变阈值时，得到了非渐近遗憾上界O(log T)。作者通过构造新的Lyapunov函数改进了Lattimore(2026a)对同一SDE的分析，并展示了用SDE工具分析策略梯度的透明性。
Convergence and Regret of the Policy Gradient for Multi-Armed Bandits in Diffusion Environment
This paper studies the policy gradient update for a multi-arm bandit problem in diffusion environment that is described by a stochastic differential equation (SDE) under the continuous-time reinforcement learning framework by Wang et al. (2020), Jia and Zhou (2022b). With the logit parameterization for the stochastic policy, we show that it converges almost surely to the optimal arm under an arbitrary constant learning rate. Furthermore, we derive the non-asymptotic regret upper bound when the constant learning rate is below a time-invariant threshold; and the regret bound has order $O(\log T)$. We improve the analysis in Lattimore (2026a) for the same SDE by constructing a novel Lyapunov function and demonstrate the transparency of analyzing policy gradient using the tools in SDEs. In addition, the same Lyapunov function is also helpful in analyzing the discrete-time policy gradient algorithm.