别光看用户点了什么赞，鼠标和眼睛动的方向才是真心话。这篇论文用59人的眼动和鼠标轨迹数据训练奖励模型，准确率从55%飙到64，还开源了数据集。
这篇论文提出利用用户与LLM交互时的鼠标轨迹和眼动数据作为隐式反馈来替代昂贵的显式偏好标注。他们构建了IFLLM数据集，包含59名用户的1336个多轮问答，并记录了鼠标和眼动数据。基于这些隐式反馈训练的奖励模型将文本奖励模型的准确率从55%提升到64%。对8个LLM应用DPO后，响应质量相对提升近3倍，证明了隐式反馈在真实场景中的价值。数据集和代码已开源。
Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Users
To align a Large Language Model (LLM), most existing methods collect explicit human feedback and train a reward model to predict the human preference based on the response text. These existing methods have two key limitations. First, the users rarely provide explicit feedback for LLM responses, which makes the high-quality preference annotation expensive to collect. Second, the methods do not leverage implicit human feedback, which has proven vital to the economic moats of Internet giants. To quantify the value of implicit feedback, we build a new dataset called IFLLM, which collects 1336 multi-turn questions from the 59 Mechanical Turk workers, their mouse trajectories, and eye gazing points to the LLMs' responses from their webcams. IFLLM shows that the users have very diverse types of gazing behavior and mouse trajectories. Our reward model based on the implicit user feedback boosts the accuracy of the text-based reward model from 55% to 64% and nearly triples the relative response quality improvements after applying the DPO to eight LLMs, demonstrating the value of implicit feedback in the wild. Our data collection website, dataset, and codes can be found at https://github.com/themehulpatwari/llm-implicit-feedback/.