研究：核机器无需特征学习也能完成 in-context learning
You Don't Need Feature Learning to Do In-Context Learning
这篇论文用核机器的谱分解把 ICL 的泛化和记忆讲成同一解的不同截断，思路很新颖。它还证明没有特征学习的模型也能做 ICL，并用高阶统计量解释了 transformer 的记忆行为。做理论研究和理解 ICL 机制的研究者应该看看。
一篇 arXiv 论文（arXiv:2611.13431）用旋转不变的核机器（kernel machines）研究 Raventos 等人 2023 年提出的 ICL 线性回归任务，在核特征基下对学习过程做谱分解。结果显示，随样本量增长，核机器先学到记忆解的低阶分量，再逐步学到依赖训练任务集高阶统计量的高阶分量。泛化解近似于记忆解的低阶截断，因此训练前期模型先泛化，继续训练反而退化为记忆。
论文给出三个结论：在 ICL 任务上学到泛化解本质上是对记忆解的欠拟合；核机器在该任务上能泛化，说明特征学习对 ICL 并非必需；作者还实证发现用 transformer 训练的 ICL 模型确实通过任务分布的高阶统计量记忆任务，与核机器理论的预测一致，说明该谱图像可能同样适用于具备特征学习能力的模型。
You Don't Need Feature Learning to Do In-Context Learning
How should we understand memorization vs. generalization in in-context learning (ICL) tasks, and when does one happen vs. the other? To investigate, we study how rotation-invariant kernel machines learn the ICL linear regression task proposed by Raventos et al. (2023). By decomposing the task in the kernel eigenbasis, we obtain a spectral picture of learning in this ICL setting. This picture reveals that as sample size grows, kernel machines learn successively-higher-degree components of the memorizing solution, which depend upon successively-higher-degree statistics of the training task set. The generalizing solution is approximately a low-degree truncation of the memorizing solution, and so the model first generalizes, then degrades into memorization as training proceeds. The takeaways are threefold. First, if this case is indicative, learning a "generalizing" solution on an ICL task is really just underfitting the memorizing solution! Second, the fact that kernel machines can generalize on this task means that feature learning is inessential to ICL. Third, transformers empirically trained to do ICL are shown to memorize tasks through the higher-order statistics of the training tasks that the kernel-based theory predicts, suggesting it applies even to models capable of feature learning.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include in-context learning, kernel machines, 谱分析, 记忆与泛化.
Source attribution: curated from arXiv cs.LG (@Joseph Turnbull, Jamie Simon).