这篇论文用条件流匹配把回报分布和多模态策略一起建模，解决了强化学习中的多模态探索难题，在DeepMind控制任务上吊打了扩散方法。
Dual-Flow RL利用条件流匹配（CFM）同时建模回报分布和多模态策略分布，克服了传统单峰高斯价值估计的偏差问题。该方法引入熵-协方差探索调节器（ECER），实现基于状态的自适应探索。在DeepMind Control Suite的36个任务中，Dual-Flow RL在32个任务上取得最优，并在Humanoid-Bench上显著优于diffusion-based和flow-based方法。ECER通过策略熵与动作不确定性协方差动态调节探索强度，避免了模式坍塌。
Dual-Flow Reinforcement Learning with State-Aware Exploration
In complex continuous-control reinforcement learning tasks, multimodal optimal actions often coincide with uncertain, multimodal return distributions, making reliable value estimation and multimodal exploration challenging. Existing value estimation methods using unimodal Gaussians restrict expressiveness and yield biased estimates. Recent generative policies can represent multimodal actions but often collapse to a few modes and under-explore high-value areas of the action space. Motivated by these challenges, we propose Dual-Flow RL, a unified actor-critic framework that jointly models a continuous return distribution and a multimodal policy distribution using conditional flow matching (CFM). This design supports reliable value estimation and sustained multimodal exploration. To further enhance exploration, we introduce an Entropy-Covariance Exploration Regulator (ECER) that enables state-aware exploration regulation leveraging policy entropy and action-uncertainty covariance. Experiments on DeepMind Control Suite and Humanoid-Bench show that Dual-Flow RL achieves state-of-the-art performance on most tasks, significantly outperforming prior diffusion-based and flow-based methods.