Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transport
This research provides a new approach to offline reinforcement learning that could improve the performance of AI systems in scenarios where data is fixed and multimodal action distributions exist.
This paper introduces OptiFlow, a framework for one-step flow policy learning in offline reinforcement learning.
It addresses challenges like mode collapse and overestimation bias by using state-wise entropic optimal transport to couple a value-aware reference flow policy with an efficient one-step policy.
The method ensures in-distribution exploitation by anchoring the one-step policy to high-value, dataset-supported modes.
Experimental results show strong performance across diverse offline RL benchmarks.
Offline reinforcement learning aims to learn a policy solely from fixed datasets, which often contain multimodal action distributions. Flow policies can naturally represent such multimodal behaviors, but learning an efficient one-step flow policy remains challenging: standard value guidance often leads to mode collapse or exploits overestimation bias in out-of-distribution regions. To address this, we introduce One-step Flow policy via Optimal Transport (OptiFlow), a framework for one-step flow policy learning as a structured sample-allocation problem. OptiFlow jointly trains a value-aware reference flow policy and an efficient one-step policy, coupling their action samples through state-wise entropic optimal transport. For each state, critic-estimated values define the priority of distillation target actions, while the action-distance cost ensures geometrically compatible pairings. By avoiding direct critic maximization, our transport-guided approach enables in-distribution exploitation by anchoring the one-step policy to high-value, dataset-supported modes without the risk of out-of-distribution divergence. Experimental results demonstrate that OptiFlow effectively captures optimal multimodal behaviors and achieves strong performance across diverse offline RL benchmarks. Our code is available at https://github.com/Yonsei-DILLab/OptiFlow.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and AI regulation, compliance frameworks, and sovereign compute policy. Key entities and technical topics under continuous tracking include optimal transport, offline reinforcement learning, multimodal, reinforcement learning.
Source attribution: curated from arXiv cs.AI (@Jaehun Shon, Jinha Choi, Jongwook Jeon, Jongmin Lee).