这篇论文提出了一个叫Patnaik-Pearson的新维度指标，用来分析BERT和DeepSeek模型内部表示的结构变化，还开源了代码，挺实用的。
该论文提出一种新的数据流形内在维度度量——Patnaik-Pearson维度，灵感来自HTSR、SETOL以及TwoNN估计器。作者证明权重矩阵经验谱密度服从帕累托分布时，该维度与HTSR和SETOL分析的尾指数临界值一致。通过理论分析和数值实验，研究了该维度在典型神经网络变换下的行为。在BERT-base和DeepSeek-R1-Distill-Qwen-1模型上，计算了token嵌入初始数据流形的维度及其随层数演变。所有代码和笔记本已在GitHub开源。
Patnaik-Pearson intrinsic dimension for internal representations of neural networks
We define a new measure of intrinsic dimension of a data manifold, which we call the Patnaik-Pearson dimension, and apply this to internal representations of neural networks, in particular transformers. The inspiration for this comes from the HTSR and SETOL work of Martin, Mahoney and Hinrichs, combined with the TwoNN intrinsic dimension estimator of Facco et al. We prove various properties of this intrinsic dimension estimator. Treating weight matrices of neural networks as data manifolds, for weight matrices whose Empirical Spectral Density follows a Pareto (Power Law) distribution, we relate the Patnaik-Pearson dimension to the HTSR and SETOL analysis, and show that critical values of the tail exponent coincide for the two approaches. Using a combination of theoretical and numerical techniques, we study the behaviour of the Patnaik-Pearson dimension of a data manifold under the transformations typical to neural networks. We apply this machinery to the BERT-base and DeepSeek-R1-Distill-Qwen-1 models, to investigate first the Patnaik-Pearson dimension of the initial data manifold of token embeddings, and second the evolution of the Patnaik-Pearson dimension as token embeddings pass through the layers of the model. Code and notebooks used for the numerical results presented here is available at https://github.com/tdhadfield/PatnaikPearson