做深度学习训练调参的开发者，这个动态p范数方案解决了ℓ₂和ℓ∞范数的极端问题，直接替换SGD就能提升泛化，值得在CIFAR/ImageNet任务上试试。
现有深度神经网络优化器多基于ℓ₂范数或ℓ∞范数，但两者各有缺陷：ℓ₂范数在高曲率方向占主导，导致低曲率方向更新缓慢；ℓ∞范数在平坦区域易振荡。本文提出动态p值的ℓ_p范数方案，融入SGD和带动量SGD，形成LPSGD和LPSGDM优化器。训练早期用大p（>2）抑制高曲率方向，后期p逐渐降至2实现稳定更新，灵感来自余弦退火策略。理论证明非凸场景下收敛率达O(T^{-1/2})，在CIFAR-10/100和ImageNet-1K上使用VGG-11、ResNet-18/50验证了更好泛化性能。
Beyond $\ell_2$-norm and $\ell_\infty$-norm: A Curvature-Inspired $\ell_p$-Norm Scheme for Deep Neural Networks
The existing optimizers for deep neural networks (DNNs) typically rely on either the $\ell_2$ norm or the $\ell_\infty$ norm, resulting in optimizers that do not adapt well to substantial changes in curvature across parameter dimensions. Generally, the training process of DNNs often exhibits strong curvature anisotropy in the early period, whereas in the later period, the training process of DNNs tends to move toward flatter regions with weaker anisotropy. Particularly, optimizers based on the \(\ell_2\)-norm are usually dominated by high-curvature directions, restricting updates of optimizers along with lower curvature direction and thus leading to a slower convergence rate. While optimizers based on the \(\ell_\infty\)-norm are prone to oscillations in flatter regions, due to the coordinate-wise updates of the same magnitude. To address these two extreme cases generated by $\ell_2$ and $\ell_\infty$ norms, we propose a novel $\ell_p$-norm scheme with a dynamical value of $p$ and incorporate it into stochastic gradient descent (SGD) and SGD with momentum (SGDM), leading to two novel optimizers with better generalization performance: ${\ell_p}$-SGD (LPSGD) and ${\ell_p}$-SGDM (LPSGDM). Particularly, the resulting optimizers suppress the dominance of high-curvature directions in the early period by utilizing a large $p$ ($p>2$), followed by a gradual decrease of $p$ toward 2 to enable more stable and refined updates, where the latter process is motivated by the cosine annealing strategy. We establish theoretical guarantees of the resulting algorithms and analyze that both LPSGD and LPSGDM achieve an \(O(T^{-1/2})\) convergence rate for the nonconvex setting. Extensive experiments are conducted on benchmark datasets, including CIFAR-10, CIFAR-100, and ImageNet-1K, with multiple DNNs such as VGG-11, ResNet-18, and ResNet-50.