这项研究揭示了语言模型层级概念几何的统计根源，对理解LLM表征形成机制的研究者很有价值，建议关注其理论框架与实验验证。
该研究提出了一种分布理论，解释语言模型中上下位关系（如“动物-狗”）的几何编码机制。基于WordNet中词对共现频率与层级距离相关的假设，理论证明word2vec嵌入的Gram矩阵谱结构会自然形成从粗到细的层级分裂几何。实验在多个WordNet子树上验证了该预测，并发现该特征在Gemma 2B模型的unembedding中同样显著。结果表明，LLM中的层级概念几何可能并非源于特定功能机制，而是词共现统计的谱结构涌现结果。
Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence
We propose a distributional theory of how hypernymy -- the ``is-a'' relation between general and specific concepts -- is encoded geometrically in language representations. Starting from the empirically verified assumption that words closer on the WordNet hypernym graph co-occur more often, we characterize theoretically the spectrum of the resulting embedding Gram matrix of word2vec embeddings. Under mild positivity and decay conditions on the co-occurrence kernel, we prove that the leading eigenvectors first separate broad taxonomic branches and then progressively finer sub-branches, producing a \emph{hierarchical splitting geometry} with a coarse-to-fine spectral organization that mirrors the tree. We confirm these predictions in word2vec embeddings across many sampled WordNet subtrees, and show that the same signature extends strikingly well to Gemma 2B unembeddings. Our results indicate that hierarchical concept geometry in LLMs need not reflect a hierarchy-specific functional mechanism, but emerges from the spectral structure of pairwise word statistics.