Papers

研究:语言模型解析词语的层数可预测人类阅读时间

The Depth of a Prediction: Where Language Models Resolve a Word Predicts Human Reading Time

Why we picked it

用 tuned lens 逐层看模型预测,提出"解析深度"这个指标,发现模型越晚确定的词人读得越慢。在 8 个眼动语料库和 GPT-2、OPT、Pythia 上验证有效。做心理语言学或可解释性研究的可以看看。

arXiv 论文提出"解析深度"(resolution depth)指标,用 tuned lens 逐层读取模型对每个词的预测,找出预测变化集中的层位。研究覆盖 8 个英语眼动与自定步速阅读语料库共 81,957 个词,以及 GPT-2、OPT、Pythia 十个模型。结果显示,模型较晚才确定词义的位置及其后文,读起来比同等困惑度的词更慢。

Body

在预测保持集阅读时间上,解析深度比最终层 surprisal、熵和最佳单层额外提升 0.26 毫奈特/词(合并 p = 0.001),约为熵增益的三分之一。对照打乱层序的深度基线,八个语料库中七个为正(随机效应均值 0.23,p = 0.016),未训练网络在 logit lens 下无增益,说明效应与真实训练表征相关。

这项工作将模型内部计算动态与人类阅读行为联系起来,为心理语言学与可解释性研究提供了跨模型家族的可复用指标。

Original · arXiv cs.LG

The Depth of a Prediction: Where Language Models Resolve a Word Predicts Human Reading Time

Surprisal from a language model's final layer predicts how long people read a word, yet the model computes its prediction across many layers. We read each word's prediction at every layer with the tuned lens and summarize this layerwise surprisal trajectory by its resolution depth, the depth around which its change is centered. On eight English eye-tracking and self-paced reading corpora (81,957 words) and ten GPT-2, OPT and Pythia models, words that a model resolves later, and the words after them, are read more slowly than equally surprising words. Resolution depth improves held-out prediction beyond final-layer surprisal, entropy and the best single layer by 0.26 millinats per word (pooled p = 0.001), about a third of entropy's gain, and each model family gains, GPT-2 and Pythia through one corpus each. The net gain varies between corpora (random-effects mean 0.07, p = 0.37); against layer-shuffled depth, which adds as many coefficients, it is positive in seven of eight corpora (random-effects mean 0.23, p = 0.016), and untrained networks show no gain under the logit lens. Pooled over corpora, the effect holds with corrected word probabilities and beyond first-to-last-layer divergences, and a whole-trajectory functional regression adds no significant information beyond resolution depth.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include tuned lens, GPT-2, Pythia, interpretability.

Source attribution: curated from arXiv cs.LG (@Donato Riccio, Fabrizio Maturo).