这篇论文用理论证明语言模型里的类内分散不是瑕疵，而是上下文信息存储，且跨14个模型验证了跨模型可预测性，很硬核。
在14个语言模型的表征分析中，类内方差并非神经塌陷的不完全，而是有特定规律的信息存储。宏观类别结构仅占表征方差的4%-12%，上下文内token占比79%-91%，该比例在100倍参数范围内稳定。理论上token级权重衰减使类别根据类型计数而非出现次数被惩罚，将next-token预测转化为不平衡K类问题。二元类别下证明逆下限：类内分散至少与条件互信息I(token; context | category)成比例。该定律跨模型和划分成立，模型间信息可预测分散程度。
Neural Collapse Is Forbidden: Information Floors in Language Models
Within-class variance in language-model representations is commonly read as incomplete neural collapse. We argue it is allocated information storage, and that the allocation obeys a law. A one-line centering identity voids a family of simplex equiangular-tight-frame claims, including our own earlier ones; in dimensionless variance shares across 14 models, macro-category structure carries only 4-12% of representational variance and within-token context carries 79-91%, stable across a 100x parameter range. On the theory side, token-level weight decay penalizes a category in proportion to its type count, not its occurrence mass, reducing next-token prediction to an imbalanced K-class problem whose optimum orders category norms by type count. A converse floor, proved for binary categories, forces within-category dispersion to be at least proportional to the conditional mutual information I(token; context | category). The law holds: identity dispersion, not total variance, tracks this information across every tested model and partition, under a model-free estimate and even across models, where one model's information predicts another's dispersion; and over pretraining the category share overshoots, decays, and partially recovers, because the information it must carry never left.