研究揭示优化器数值精度墙：Adam 在幂律谱低秩矩阵感知中的失效边界
The Precision Wall: An Optimizer-Dependent Boundary for Optimization under Power-Law Measurement Spectra
一篇优化器理论论文：解释了 Adam 在幂律测量谱下为什么会莫名卡住，给出了能预测失效起点的公式，做低秩优化的可以看看。
论文研究 rank-D 低秩矩阵感知，测量协方差从各向同性高斯换成幂律谱 diag(j^{-s})。分析表明优化器存在精度墙：Hessian 最小特征值按 V^{-(s+1)}D^{-s} 缩放，一旦低于优化器 ε_O 的分母精度下限，归一化噪声会淹没稀有方向梯度。作者推导出临界指数边界公式 s*_O(V,D)，用 4 组配置标定 Adam 的 δ 后冻结，在 16 格网格（V∈{64,128,256,512}×D∈{4,8,16,32}）上预测 Adam 失效起点误差不超过 0.45。该公式还能区分分母精度墙、条件数墙和符号量化墙，但在 D=2 处失效，且在 ε_0=10^{-8} 以下触及动力学下限。
The Precision Wall: An Optimizer-Dependent Boundary for Optimization under Power-Law Measurement Spectra
In low-rank matrix sensing of a rank-$D$ factor $U^\ast \in \mathbb{R}^{V \times D}$, replacing isotropic Gaussian designs with power-law measurement covariance $Σ= \mathrm{diag}(j^{-s})_{j=1}^V$ leaves the Burer--Monteiro landscape free of spurious local minima, yet division-normalised first-order optimizers stall above a critical exponent $s^\ast$. We trace this failure to a numerical precision wall. At the ground truth, the smallest non-gauge expected Hessian eigenvalue scales at leading order as $V^{-(s+1)}D^{-s}$. When this curvature drops below the effective denominator floor $ε_O$ of optimizer $O$, normalisation noise overwhelms rare-direction gradients, yielding the leading-order boundary $s^\ast_O(V, D) = [\log(1/ε_O) - \log V]/\log(VD) + δ_O$, where $δ_O$ is a metric-dependent detection lag. With $δ_{\mathrm{Adam}}$ calibrated on four configurations and frozen, this formula predicts Adam's onset within $0.45$ across all 16 cells of a held-out grid ($V \in \{64,128,256,512\} \times D \in \{4,8,16,32\}$), separates fixed-$VD$ pairs, and distinguishes denominator-precision walls from conditioning and sign-quantisation walls. The approximation fails at $D = 2$, reflects a finite-$D$ spectral crossover, and saturates below $\varepsilon_0 = 10^{-8}$ at a dynamical floor.