Ridge 回归中迁移学习的极小最优 Scaling Law 证明
Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning
一篇 arXiv 理论论文，证明了 ridge 回归迁移学习何时能达到极小最优 scaling law，还顺手证明单遍 SGD 就够用。
论文研究预训练源模型通过 ridge 回归训练、目标模型以 L2 距离正则化微调的两步迁移学习流程，在高维回归设定下推导出风险的确定性等价。在共享回归函数与幂律协方差谱的设定下，作者识别出一个 scaling law 达到极小最优指数的区域。该区域内最优源正则化只依赖源数据，对任意目标任务通用。论文还证明用单遍 SGD 做微调可达到同样最优速率，并在合成数据和语言模型实验中验证了结论。
Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning
Pretrained models are often reused across downstream tasks, motivating the question of whether this transfer learning procedure can provide a substantial performance improvement, namely, in terms of the scaling law. We address this question in the prototypical setting of high-dimensional regression, where (i) a pretrained \emph{source} model is trained via ridge regression, and (ii) a fine-tuned \emph{target} model is regularized via the $L_2$ distance from the source estimator. First, we establish a deterministic equivalent for the risk of this two-step procedure under both model and covariance shift. Next, by specializing the deterministic equivalent to a shared regression function and commuting power-law covariance spectra, we identify a regime where the resulting scaling law achieves the minimax-optimal exponent. Notably, in this regime, the optimal source regularization depends on source data only, indicating its universality across all targets. Furthermore, we show that the same optimal rate is achieved by performing the fine-tuning step via one-pass stochastic gradient descent. We support our theoretical analysis via experiments on both synthetic settings and language models.
