PreferenceEKF 用子空间推理解决奖励模型不确定性问题，比传统贝叶斯方法更高效。
研究团队提出 PreferenceEKF 方法，通过扩展卡尔曼滤波器在低维参数子空间中进行顺序推理，解决了大型神经网络奖励模型的不确定性量化难题。该方法在 D4RL 和 V-D4RL 基准测试中展现出比其他贝叶斯深度学习方法更好的样本效率、运行时间、可扩展性和校准性能。学习到的奖励模型在离线强化学习策略性能方面具有竞争力。
Subspace Inference Enables Efficient Active Reward Learning from Preferences
Reinforcement learning from human feedback (RLHF) has emerged as a powerful yet sample-inefficient approach for learning reward models from human preferences, making active learning a critical component in synthesizing informative preference queries. However, effective uncertainty quantification required for active learning remains a key challenge for large neural network reward models. In this paper, we introduce PreferenceEKF, a sample-efficient approach that tracks reward model uncertainty by framing active preference learning as a sequential Bayesian filtering problem. Instead of relying on computationally prohibitive posterior inference over the full neural network parameter space, our method performs sequential inference via an extended Kalman filter within a low-dimensional parameter subspace, continuously updating the reward model posterior as new preference queries arrive. Our approach enables scalable sampling of neural network parameters to efficiently compute acquisition functions for active reward learning. Experiments on the D4RL and V-D4RL benchmarks demonstrate that our approach achieves better sample efficiency, runtime, scalability, and calibration compared to other Bayesian deep learning approaches, and the learned reward models lead to competitive offline reinforcement learning policy performance. This highlights the potential of scalable Bayesian methods for preference-based reward modeling in RLHF. Our code is available at https://github.com/yutaizhou/bnn_pref.