这篇论文为预训练数据规模定律提供了首个理论证明，做元学习或预训练研究的学者值得关注，可以直接参考其复杂度正则化方法改进现有模型。
该论文提出了复杂度最小化框架，用于理论解释预训练数据规模增大如何降低下游样本复杂度。通过端到端分析，证明该框架能捕捉元学习中的缩放行为，即少样本适应误差率随元训练数据量增加而改善。实验表明，将复杂度正则化融入现有元学习方法能持续提升下游样本效率。
Provable Data Scaling Law for Meta Learning via Complexity Minimization
Pre-training has become a fundamental paradigm in modern machine learning, with one of its key empirical benefits being reduced downstream sample complexity as the scale of pre-training data increases. However, existing theoretical frameworks for pre-training do not fully explain this phenomenon. In this paper, we introduce complexity minimization, a novel meta-representation learning framework designed to enable theoretical analysis of this scaling behavior, which learns representations by evaluating the downstream model complexity best suited to each domain and minimizing the worst-case such complexity across source domains. Our end-to-end theoretical analysis, spanning pre-training through downstream regression, shows that this framework provably captures this scaling behavior; in particular, we show that the error rate of few-shot adaptation improves as the amount of meta-training data grows. Empirically, we demonstrate that incorporating complexity regularization into existing meta-learning methods consistently improves downstream sample efficiency.