做机器人控制或连续控制RL的团队，如果受困于扩散/流模型训练的不稳定性，QGF提供了一种“训练照旧、测试优化”的实用方案，值得一试。
强化学习中的扩散/流模型策略虽在模仿学习中表现出色，但整合到RL训练中常因稳定性问题而受限。本文提出QGF（Q-Guided Flow），一种仅在测试时进行策略优化的RL算法。它预训练一个参考流策略（通过行为克隆）和一个价值函数，在测试时利用价值梯度引导参考策略生成更高价值的动作，无需额外策略学习。实验表明，QGF在离线RL基准上优于现有测试时方法，与最先进的训练时算法性能相当且计算成本更低，且随模型规模扩展表现良好。
Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning
Expressive continuous control policies, such as diffusion and flow models, form the backbone of recent advances in scaling imitation learning for simulated and real robot control. While they are known to scale stably in the supervised imitation learning setting, incorporating them into reinforcement learning (RL) pipelines for policy improvement has proven more difficult. It often requires specialized training objectives or backpropagating through denoising processes, which cause well-known issues with stability and affect scalability. In this paper we study the question of whether simple policy improvement schemes at test time alone, leaving stable supervised policy training intact, can be a competitive alternative which sidesteps these issues. To this end, we propose QGF (Q-Guided Flow), an RL algorithm that performs policy optimization entirely at test time. QGF works by pre-training both a reference flow policy (via a standard behavioral cloning objective) and a value function critic and, at test time, using the value gradient to guide the reference policy to generate higher-value actions without any additional policy learning. Empirically, QGF outperforms prior test-time RL methods on single-task and goal-conditioned offline RL benchmarks with high-dimensional action spaces, and is competitive with state-of-the-art training-time algorithms while being much cheaper to run. Moreover, it exhibits favorable scaling with model size by avoiding the instability of actor-critic training, offering a practical and effective alternative RL algorithm with expressive policies.