部分可训练神经网络近似误差出现相变
Phase Transition from Random Features to Fully Trainable Networks
论文算出：只训练不到 5% 的神经元就能拿到几乎全部的近似精度提升，做科学计算或网络压缩的可以看看这条数学依据。
arXiv 论文研究随机特征与全可训练网络之间的部分可训练神经网络。在多项式谱衰减指数 α>1 与 leverage 兼容性条件下，论文证明宽度 m 的两层网络的平方 L2 近似速率为 O((κ_P log(m/δ)/m)^α)。对于维度 d=5、初始化尺度 τ=10 的球形偏置 ReLU 特征，仅约 4.1% 的可训练比例就能获得从 P=0 到 P=1 近似界缩减量的 90%。数值实验覆盖最多 15 层的网络，训练误差呈现相同的先快速后收益递减的相变模式。
Phase Transition from Random Features to Fully Trainable Networks
Constructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing. Partially trainable neural networks bridge fixed random features and fully adaptive representations. This paper quantifies a phase transition pattern in approximation bounds and observes analogous behavior in training errors: a rapid but smooth initial improvement followed by diminishing returns, with a sharper transition under greater initialization mismatch. Under polynomial spectral decay with exponent $α>1$ and leverage compatibility, we establish a squared $L_2$ approximation rate $\mathcal{O}((κ_P\log(m/δ)/m)^α)$ for two-layer networks of width $m$ and trainable proportion $P$, where $κ_P=(P/κ_1^*+(1-P)/κ_0^*)^{-1}$. The bound holds uniformly over the unit ball of a reproducing kernel Hilbert space (RKHS) with probability at least $1-δ$. The constants $κ_0^*$ and $κ_1^*$ quantify the fixed and trainable groups' sampling mismatches relative to the target RKHS. For spherical biased-ReLU features in dimension $d$, we verify these assumptions with $α=(d+2)/(d-1)$ and $κ_0^*=4\sqrt{2}τ(d+1)/d$ for initialization scale $τ\ge1$, and $κ_1^*=1$ under ridge leverage sampling. For $d=5$ and $τ=10$, a trainable proportion of approximately $4.1\%$ captures $90\%$ of the reduction in the approximation bound from $P=0$ to $P=1$. A structured three-layer example illustrates how a few trainable first-layer neurons exploit fixed downstream blocks through nonlinear composition. Numerical experiments across diverse settings, including networks with up to 15 layers, exhibit similar phase transitions in training errors.
