Papers

MPG-DPO 用多维偏好数据降低对齐标注成本

Sample-efficient Personalization of Large Language Model Alignment with Multidimensional User Preference Data

Why we picked it

一篇偏方法的论文,思路是把自动验证器的多维评分变成用户专属的"偏好锥"来筛对比样本,省掉大量无效标注。做在线 DPO、RLHF 或个性化对齐的工程师可以直接看它的样本筛选逻辑,多个 LLM 家族上都验证了效果。

arXiv 论文 2611.22210 提出 MPG-DPO(multidimensional preference-guided online direct preference optimization),针对在线偏好对齐中人工标注昂贵、样本效率低的问题给出新方案。现有方法从 LLM 视角度量不确定性来挑选对比样本,但用户的真实偏好可能早已排除某些选项,导致标注浪费。

Body

MPG-DPO 的做法是利用自动化验证器对用户历史对比数据生成低成本低维评估,构建一个个性化的"偏好锥",先丢弃在用户偏好下被明确支配的响应,再从剩余候选中挑选竞争性强且分歧最大的响应对进行标注。理论分析证明,随学习推进,浪费在被支配响应上的查询比例会趋近于零。

实验覆盖多个 LLM 家族,在相同查询预算下 MPG-DPO 优于强基线方法。对企业团队而言,这意味着用更少的人工反馈完成个性化对齐,直接降低在线 RLHF 流程的标注成本。

Original · arXiv cs.LG

Sample-efficient Personalization of Large Language Model Alignment with Multidimensional User Preference Data

Preference alignment has become a central tool to align large language models (LLMs) with human values and desired behavior, and its online variant can further advance their capabilities beyond what static data alone offer by repeatedly generating and labeling new responses with human feedback. But costly, time-consuming annotations limit its scalability, making sample efficiency essential, and existing methods typically seek to improve it by expanding coverage of the response space or selecting pairs with the highest uncertainty for comparison. Although they are effective, these methods quantify uncertainty from the perspective of the LLM, which may differ from the uncertainty perceived by the user: a pair can appear highly uncertain to the LLM even when past comparisons from the user already imply a clear ordering. In this paper, we propose multidimensional preference-guided online direct preference optimization (MPG-DPO), which leverages low-cost multidimensional evaluations from automated verifiers on the user historical comparisons to construct a personalized preference cone guiding response selection for online preference alignment. Specifically, it first discards responses that are confidently dominated under the estimated user preference, then selects two responses that are competitive under the retained preference cone yet exhibit the strongest unresolved disagreement. Theoretical analysis shows that, with high probability, the fraction of queries spent on cone-dominated responses vanishes as learning progresses. Experiments across multiple LLM families show that MPG-DPO outperforms strong contenders under matched query budgets.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include DPO, preference alignment, RLHF, personalization.

Source attribution: curated from arXiv cs.LG (@Zhen-Yu Zhang, Jerry Huang, Gang Niu, Takashi Ishida, Masashi Sugiyama).