UBP2通过主动探索和不确定性平衡，有效解决了偏好强化学习中样本效率低的问题。在Meta-World测试中效果显著。
UBP2是一种基于模型的偏好强化学习方法，通过联合推理奖励、动态和价值函数的不确定性来主动引导探索。该方法使用集成模型对候选轨迹进行评分，平衡期望奖励、终止价值和认知不确定性。在Meta-World基准测试中，UBP2比无模型的偏好方法和非乐观的基于模型基线实现了更高的样本效率。
UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning
Preference-based RL provides an approach to learning reward models from pairwise comparisons of behaviors, bypassing the need for explicit reward design. However, existing methods typically rely on passive data collection and suffer from poor sample efficiency, especially during the early stages of learning. We introduce a model-based approach that actively directs exploration by jointly reasoning over uncertainties in the reward, dynamics, and value functions. Our method, Uncertainty-Balanced Preference Planning (UBP2), uses ensembles of reward, dynamics, and value function models to evaluate candidate trajectories according to a unified score that combines expected reward, terminal value, and epistemic uncertainty. Planning under this objective yields an explicit tradeoff between exploitation and information acquisition without requiring ad hoc exploration heuristics. Under standard regularity assumptions, we establish sublinear regret guarantees for both finite-horizon and infinite-horizon settings. Empirically, experiments on the Meta-World benchmark show UBP2 achieves substantially higher sample efficiency than model-free preference-based methods and non-optimistic model-based baselines.