大语言模型中的影评取向:来自电影偏好诱导的证据

Critical Acclaim Orientation in Large Language Models: Evidence from Film Preference Elicitation

精选理由

各家大模型选电影时都更认影评口碑,不认票房。测试了20万次对比,模型越大越看重口碑,挺有意思的。

AI 摘要

一项研究分析了八种来自Anthropic、OpenAI、阿里和Mistral四家公司的模型在电影评价中的取向。研究者使用200部电影基准,将其划分为影评口碑、商业成功和双重认可三类,对每个模型进行20,000次两两强制选择。基于Bradley-Terry估计发现,所有模型都更偏好影评口碑好但商业上不知名的电影,而非商业成功但影评认可低的电影,且该取向随模型规模增大而增强。嵌套OLS回归显示,评价取向、公开可见度和大众接受度分别影响偏好;调整公开可见度后,模型对双重认可电影的偏好发生逆转。评价型与推荐型提示词会引发不同排序,显示该取向可能在真实部署中间接显现。

原文 · arXiv: Anthropic

Critical Acclaim Orientation in Large Language Models: Evidence from Film Preference Elicitation

Large language models (LLMs) are trained on corpora that contain expressions of human judgment about films, books, music, and more. Yet whether LLMs systematically reproduce evaluative hierarchies remains unclear. Prior research on cultural bias in LLMs suggests competing expectations: models may mirror the popularity signals of internet texts, or may reproduce forms of prestige embedded in critical discourse. We probe this question through a study of film evaluations with eight models from four families (Anthropic, OpenAI, Alibaba, and Mistral), using a 200-film benchmark partitioned into critically acclaimed, commercially successful, and dual-legitimacy (critical acclaim + commercial success) films. Across 20,000 pairwise forced-choice comparisons per model analyzed with Bradley--Terry estimation, we observe a consistent critical acclaim orientation with all models: critically acclaimed yet commercially obscure films are selected over commercially successful yet critically unrecognized ones. This pattern grows with model scale within each family. In addition, nested OLS regression analyses show that evaluative orientation, public visibility, and popular reception distinctly help explain preferences. Adjusting for public visibility reverses the models' preference for dual-legitimacy films over critical acclaim-only films, while additionally accounting for popular reception attenuates much of the disadvantage of films with commercial success only. Finally, evaluative and recommendation-oriented prompt framings produce divergent rankings, suggesting that critical acclaim orientation may manifest indirectly in real-world LLM deployments.

大语言模型中的影评取向:来自电影偏好诱导的证据 · AI 热点