临界阻尼 Polyak 步长:让梯度下降收敛到更平坦极小值的新算法
A new algorithmic approach for finding flat minima: critically damped Polyak stepsizes
搞训练的朋友可以看看这篇:不用 SAM 的两次前向,一个改过的 Polyak 步长就能压低 sharpness,连 LLM 微调都验证过,还带一致时间收敛证明。
arXiv 论文提出一种新的自适应步长规则——临界阻尼 Polyak(CDP)步长,在经典 Polyak 规则基础上乘以系数 4 并引入阻尼因子,使梯度下降偏向更平坦的极小值。作者首次对一大类局部二次目标给出了 sharpness 与损失下降的一致时间保证。
机制上,系数 4 会在极小值流形 M 附近产生等振震荡,阻尼项则驱动迭代收敛到 M。CDP 通过适应局部曲率运行在稳定边缘:损失较小时,其投影迭代近似沿 M 上的 log λ₁(∇²L) 做梯度下降,与 Arora 等人 2022 年分析的归一化梯度下降的平坦性偏好类似。区别在于 CDP 的步长随到 M 的距离缩短,能在有限学习率下收敛到局部最小化 sharpness 的插值解。
实证方面,CDP 产生了与 Sharpness-Aware Minimization(SAM,Foret 等,2021)类似的平坦化效果,且在理论范围之外也有效,包括非线性 teacher-student 网络和预训练 LLM 的微调。对需要无需额外前向开销即可降低 sharpness 的训练方来说,这是 SAM 之外的另一个选项。
A new algorithmic approach for finding flat minima: critically damped Polyak stepsizes
We present a new adaptive stepsize rule that biases gradient descent toward flatter minima in both theory and practice, and establish the first uniform-in-time guarantees for both sharpness and loss reduction for a broad class of locally quadratic objectives. Our critically damped Polyak (CDP) stepsizes multiply the classical Polyak rule by $4$ and a mild damping factor. The factor $4$ is critical in that it produces equioscillations around the manifold of minimizers $\mathcal{M}$ to leading order. Damping provably drives convergence to $\mathcal{M}$. CDP thus operates at the edge of stability by adapting its stepsizes to local curvature. Once the loss is small, its projected iterates follow gradient descent on $\log{λ_1(\nabla^2 \mathcal{L})}$ along $\mathcal{M}$ to leading order, paralleling the flatness bias of normalized gradient descent (Arora et al., 2022). Unlike normalized gradient descent, CDP step lengths shrink with the distance to $\mathcal{M}$, enabling convergence to an interpolating solution that locally minimizes sharpness at finite learning rate. We establish our theory for a broad class of non-negative losses with locally quadratic Morse-Bott structure near $\mathcal{M}$, capturing regular interpolation regimes in smooth overparameterized neural networks. Empirically, CDP stepsizes produce a flattening effect similar to that of Sharpness-Aware Minimization (SAM, Foret et al., 2021), reducing sharpness even in settings beyond our theory, including nonlinear teacher-student networks and fine-tuning of pre-trained LLMs.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include SAM, gradient descent, optimization algorithms, sharpness.
Source attribution: curated from arXiv cs.LG (@Francesco Caporali, Jason M. Altschuler, Boris Hanin).