Negative-Shifted Gradient Descent突破负脊端点,实现混合符号谱正则化

Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent

精选理由

这篇论文揭示了负移位梯度下降的谱正则化机制,能突破负脊端点的结构限制,在过参数化线性回归中实现更优的混合符号滤波。

AI 摘要

本文在过参数化线性回归中提出负移位梯度下降(NSGD),其滤波器在负脊端点处光滑,支持混合符号滤波。在Gaussian spike-plus-flat模型中发现了Marchenko-Pastur barrier:移位量需位于最小经验特征值之上一个bulk宽度,停止路径对比所有可接受的端点风险改进多项式因子。主要定理适用于高有效秩尾部,通过局部Duhamel积分控制非收缩移位动力学。有限格留出不等式将分离转移到验证选择算法。

原文 · arXiv cs.LG

Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent

In overparameterized linear regression, many weak spectral directions act like a ridge penalty on the signal-bearing spectrum; negative ridge is the natural correction, pushing filters above one. The stable negative-ridge endpoint, however, is structurally limited: its pole must stay below the smallest nonzero empirical eigenvalue, and it anti-shrinks smaller eigenvalues more than larger ones. Early-stopped negative-shifted gradient descent escapes this constraint. Its filter is smooth at the would-be pole and mixed-sign-capable: above-ridgeless directions form a leading prefix, with lower directions shrunk or exposure-controlled while stopping sets the crossover. In a Gaussian spike-plus-flat model we discover a Marchenko-Pastur barrier: the shift that cancels the implicit penalty lies a bulk width above the smallest empirical eigenvalue, and the stopped path improves on every admissible endpoint by a polynomial factor in risk under explicit conditions. Our main theorem permits a general high-effective-rank tail: its trace sets the implicit floor, its squared spectrum controls exposure, and the floor-critical path recovers all head scales at once, beyond positive shrinkage and, once scales separate, every uniform rescaling of ridgeless. Handling the noncontractive shifted dynamics is the central technical challenge; localized Duhamel integrals control them. A finite-grid hold-out inequality transfers the separations to the validation-selected algorithm.