如果你做强化学习微调，这篇论文告诉你预训练Q函数可能没用，还给了更高效的IPE方法，实测效果更好。
论文研究了在线RL微调时预训练Q函数的效果。发现预训练Q函数相比随机初始化提升有限，因为预训练的Q函数目标与在线微调收敛的Q函数存在根本性不匹配。作者提出策略集成初始化（IPE）方法，通过训练多个多样策略并用其滚动数据引导Q函数学习。在连续控制基准上，IPE相比朴素Q函数预训练平均提升1.26倍微调性能。
Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?
Pre-training followed by fine-tuning has become the dominant recipe for learning performant policies, and in value-based reinforcement learning (RL) this raises a natural question: given a pretrained policy, should the Q-function be pretrained on offline data too? Conventional wisdom suggests it should, but recent results show that online RL with a randomly-initialized Q-function can result in highly performant and reliable policies without needing to pretrain the Q-function. In this paper, we systematically study whether pretraining the Q-function actually helps when fine-tuning on top of a pretrained base policy. We find, surprisingly, that naive Q-function pretraining often provides little benefit over random initialization. We show this stems from a fundamental mismatch: the Q-function learned during pretraining targets the pretrained policy's Q-function, not the Q-function that online fine-tuning converges to, and this gap persists even after offline value maximization. Motivated by this finding, we propose Initialization via Policy Ensemble (IPE), a simple method that trains multiple diverse policies and uses their pooled rollouts to bootstrap the Q-function learning in online RL. Across a suite of challenging continuous control benchmarks, IPE yields an average 1.26x improvement in fine-tuning performance over naive Q-function pre-training.