注意力对大小写敏感：LLM内隐的排版注意力机制
Attention is Case-Sensitive
这篇arXiv研究测了13个模型：把关键词大写能集中LLM注意力，但准确率不一定提高，交替大小写还会拖累成绩。
该研究在13个模型（9个LLM和4个VLM）中系统评测了字母大小写对注意力分配的影响。实验发现，将目标信息改为交替大小写或大写，能显著集中文本注意力。
但这种注意力集中并不提升下游准确率，在交替大小写的高熵场景下甚至会造成下降。推理模型的思考阶段会缓冲这类排版敏感性，而视觉语言模型则表现出部分迁移效应。
研究将大小写视为无需模型访问或微调的零样本注意力引导机制。
Attention is Case-Sensitive
In human visual perception, uppercase lettering serves as a natural salience cue that captures attention within lowercase text. In this paper, we present a systematic empirical characterization study revealing that Large Language Models (LLMs) exhibit an analogous property: letter casing modulates internal attention allocation. Through analysis across 13 models, nine LLMs and four Vision-Language Models (VLMs), with diverse tokenization schemes, we show that formatting target information in alternating or uppercase against a lowercase context concentrates attention on those textual spans. In text this effect is universal, holding across every evaluated non-reasoning model. We frame it as a previously under-explored latent property of pretrained transformers rather than a prescriptive method. Our investigation reveals a central attention-performance divergence: while this "casing effect" robustly shifts attention, its impact on downstream accuracy is non-trivial, increased concentration does not inherently improve task accuracy and, in high-entropy contexts like alternating case, can degrade it. We further identify a boundary condition: the deliberative "thinking" phase in reasoning models acts as a semantic buffer that mitigates typographic sensitivity in text. Extending the study to VLMs, we find the effect transfers partially: the same prompt-side casing reorganizes cross-modal attention along two coupled axes, predominantly a macroscopic disengagement from the image toward the text prompt, and secondarily a concentration of the residual visual attention on the target region. By isolating casing as a zero-shot mechanism for attention steering that requires no model access or fine-tuning, we provide a new foundational understanding of how pretraining internalizes typographic emphasis.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include 大小写, 注意力机制, LLM, 视觉语言模型.
Source attribution: curated from arXiv cs.LG (@Maximilian Dillitzer, Tin Stribor Sohn, Jason J. Corso, Michael Auerbach).
