这篇论文把PPO用到了GFlowNet上，收敛更快、数据效率更高，做离散采样研究的可以看看。
该论文将近端策略优化（PPO）应用于生成流网络（GFlowNet）框架下的离散概率分布采样。作者推导了GFlowNet的等价策略梯度算法，并实验探索了基线训练和优势估计等环节。这是首次成功将PPO应用于GFlowNet，在合成能量和分子图生成等基准上，PPO相比标准GFlowNet训练目标提升了收敛速度和数据效率。
This paper explores policy gradient algorithms for training stochastic policies to sample from structured discrete probability distributions under the Generative Flow Network (GFlowNet) framework. Building on extensive theoretical connections between GFlowNets and entropy-regularized reinforcement learning, we derive equivalents of standard policy gradient algorithms for training GFlowNets, as well as experimentally explore their various methodological aspects, including baseline training and advantage estimation. Most importantly, our work is the first to derive and successfully apply proximal policy optimization to GFlowNets, showing its improved convergence speed and data efficiency compared to standard GFlowNet training objectives on benchmarks ranging from synthetic energies to molecular graph generation.