PrefMind：双智能体在线偏好学习框架改进多模态生成
Beyond Reward Scoring: Online Multi-Agent Preference Learning for Multimodal Response Generation
这篇论文把偏好理解从打分器搬进了生成模型内部，Generator 和 Critic 两个智能体互相博弈，三个模型规模都验证有效。
arXiv 论文提出 PrefMind，一种在线双智能体强化学习框架，包含响应 Generator 和偏好 Critic 两个可训练智能体。两个智能体针对候选响应对分析讨论哪个更符合人类偏好，人类标签监督其判断与修正。当双方正确识别人类偏好响应时，Generator 生成改进答案，Critic 将其与原偏好响应比较并给出生成奖励。在三个模型规模上的实验显示，偏好判断和下游响应生成均有提升。
Beyond Reward Scoring: Online Multi-Agent Preference Learning for Multimodal Response Generation
Multimodal preference alignment often follows a score-based RLHF pipeline: a reward model is first trained to learn human preferences, then used to score on-policy responses and optimize the response model's generation. This turns preference supervision into a scalar signal for generation: a model may learn to produce answers with higher preference scores, but is rarely trained explicitly to learn the reasons underlying human preferences. We therefore ask whether making preference understanding an internal capability of the response model can improve its generation. To investigate this question, we propose \textbf{PrefMind}, an online dual-agent reinforcement learning framework with two trainable agents: a response Generator and a preference Critic. Given a pair of candidate responses, the agents analyze and discuss which response better aligns with human preference. Human labels supervise their judgments and corrections. For pairs on which both the Generator and revised Critic correctly identify the human-preferred response, the Generator produces an improved answer, and the Critic compares it against the original preferred response to provide a generation reward. Both agents are jointly updated from interactions sampled with their current policies. Through this process, \textbf{PrefMind} trains the Generator to understand human preferences via explicit judgment and correction, and to translate this understanding into better responses. Experiments across three model scales show improvements in both standalone preference judgment and downstream response generation.
