这是篇新论文，LEMUR能直接从多人偏好学多目标策略，不用设奖励函数，基准上比现有方法强。
LEMUR是一个新框架，让智能体从多个人的偏好反馈中学习多目标策略，无需预先定义奖励函数。它同时学习策略和多个目标专属的奖励模型，在训练中平衡竞争目标。在多种多目标任务基准上，LEMUR的表现优于现有基线方法。这项研究为无预设奖励的多目标决策任务提供了新方向。
LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback
Reinforcement Learning (RL) systems are typically trained using a single, well-specified scalar reward function. However, real-world decision-making tasks often involve multiple, competing objectives, such as performance versus efficiency, where ground-truth reward functions are difficult to specify or inaccessible. While Multi-Objective RL (MORL) addresses such trade-offs by modeling rewards as vectors, existing approaches typically assume access to a well-specified reward function for each objective, inheriting the same challenges faced by single-objective RL. Meanwhile, Preference-based RL (PbRL) has shown great potential in solving complex tasks without access to a pre-defined reward function through reward learning from human feedback, yet has largely been studied in single-objective settings. In this work, we bridge this gap with LEMUR: Learning to Align with Multi-Objective Reinforcement Learning with Preference feedback, a novel framework where an agent interactively learns from the preferences of multiple humans to learn optimal multi-objective policies. Our approach jointly learns policies and multiple objective-specific reward models from human feedback, enabling agents to effectively balance competing objectives during learning. We evaluate LEMUR on a variety of benchmark multi-objective tasks, and empirical results demonstrate its superior performance over baseline methods. Our method presents a promising direction for solving multi-objective decision-making tasks without pre-defined reward functions.