RefineICL 论文解读：表格基础模型如何在上下文内细化表示
What Do Tabular Foundation Models Compute In Context? In-Situ Representation Refinement through Attention-Gated Updates
这篇把表格模型上下文学习拆开讲了，RefineICL 比 TabPFN-3 高 31.4 Elo，还解释了为什么去掉 FFN 反而更好，做表格任务的建议看看。
论文提出 in-situ representation refinement 机制，让表格基础模型在推理时用支持集标签更新表示并迁移到无标签查询，且不改变模型参数。基于此设计的 RefineICL-L24 在 AMLB29 上达到 0.93836 OVR-AUC 和 0.87173 准确率，在 TabArena 38 数据集快照上拿到 1644.8 Elo，比 TabPFN-3 高 31.4 Elo。消融显示在匹配的 100K 更新深度下，加入 FFN 无一致收益，还在 L8 层多耗 60.2% 推理峰值内存。内部干预实验覆盖全部 72 个测试 episode，证明支持集更新不只是静态标签来源。
What Do Tabular Foundation Models Compute In Context? In-Situ Representation Refinement through Attention-Gated Updates
What reusable computation should a tabular foundation model learn when every table defines a new supervised task? We develop in-situ representation refinement: support labels guide updates to the episode's representations, and these updates transfer to unlabeled queries without changing model parameters. A regularized leave-one-out objective yields a support correction and its query extension. The leading term separates attention-based reading from state-dependent scaling, motivating RefineICL: an attention-gated, FFN-free contextual stack with selected low-rank feature interaction and typed memory. RefineICL-L24 reaches 0.93836 OVR-AUC and 0.87173 accuracy on AMLB29. A benchmark-informed continuation reaches 1644.8 Elo on the 38-dataset TabArena snapshot, 31.4 Elo above TabPFN-3 under the same evaluation. It also improves all four reported metrics over TabPFN-v3 on both TabZilla views. In a matched 100K-update depth grid, an expanded FFN gives no consistent validation benefit and uses 60.2% more peak inference memory at L8. Internal interventions show that support representations are more than a static source of labels: removing one intermediate support update, while preserving the query output, increases final query cross-entropy in all 72 tested episodes. Together, the derivation and interventions explain how attention-gated updates can construct a task-specific predictor in context.
