利用人类注视分布数据训练奖励模型提升偏好预测准确率
Leveraging Psychophysical Attentional Distribution for Gaze-Augmented Reward Modeling
训练奖励模型时只盯一个注视点太粗糙了，这篇论文把注意力做成空间梯度分布，准确率确实涨了。做 RLHF 或对齐研究的同学可以看看这个建模思路。
一篇 arXiv 论文提出把眼动追踪数据引入 RLHF 奖励模型训练，并改进了以往只关注注视点的方法。研究者借鉴认知心理学关于人类字母识别的结论，将视觉注意力建模为以当前注视位置为中心的空间梯度分布，再把这个表示加入奖励模型训练。
结果显示，采用该注视分布训练的奖励模型在偏好预测准确率上高于基线模型，部分实验结果还与阅读中人类注意力分配的既有认知研究发现定性一致，说明模型可能捕捉到认知上有意义的注意力分配特征。
Leveraging Psychophysical Attentional Distribution for Gaze-Augmented Reward Modeling
Integrating human feedback into language models has gained increasing attention as a means to align model outputs with human preferences, with Reinforcement Learning from Human Feedback (RLHF) serving as a prominent framework for preference-based alignment. Recent studies have incorporated eye-tracking (ET) data as an additional supervisory signal for reward model training, grounding reward learning in human reading behavior. Building on this approach, we extend prior fixation-centric models by leveraging established findings from cognitive psychology on human letter recognition. Specifically, we model visual attention as a spatially graded distribution centered on the current gaze location and incorporate this representation into reward model training. Our results show that the reward model trained with the proposed gaze distribution achieves higher preference prediction accuracy than a baseline model. Moreover, some results from our experiments are qualitatively consistent with established findings on human attention during reading, suggesting that they potentially capture cognitively meaningful aspects of human attention allocation.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include RLHF, 眼动追踪, 奖励模型, 人类反馈对齐.
Source attribution: curated from arXiv cs.LG (@Chaeho Lee, Junyup Kim, Wansoo Kim, Young Min Jung, Sang Ho Lee).
