Papers

Lie bracket 或成灾难性遗忘早期预警信号:无需旧数据即可预测

Does Learning Order Explain Forgetting? The Lie Bracket as an Early-Warning Signal for Catastrophic Forgetting Without Old Data

Why we picked it

用一阶 Lie bracket 预测遗忘,这个想法很新颖,而且实测只需要存梯度和 20 个 Hessian 特征对,不用保留旧数据,比 EWC 的 rho 高出不少。做持续学习的人可以看看它的适用边界,快速遗忘场景它还是打不过 EWC。

论文实证检验了 Dherin (2023) 的猜想:两个任务梯度场的 Lie bracket 可能与灾难性遗忘相关。在 Split MNIST、Split Fashion-MNIST 和 Split CIFAR-10 三个基准上,任务 A 的遗忘量几乎精确等于两种训练顺序下 A 损失之差,Spearman rho 达 0.96-1.00。

Body

关键实验只需在任务 A 结束时存储其梯度和 top 20 Hessian 特征对,即可在不接触 A 数据的情况下、在任务 B 训练早期预测遗忘。当任务 A 学得充分且遗忘缓慢(task-incremental 场景)时,bracket 排序遗忘的相关性达 rho = 0.70-0.81,高于 Elastic Weight Consolidation 惩罚项的 0.33-0.58,并达到或超过在 A 原始数据上重新评估的效果。

该信号有明显边界条件:domain-incremental 等快速遗忘场景下存储摘要会失效,EWC 惩罚反而更好;旧任务只学到一半时,A 上残留梯度比正常情况大一个数量级,一阶损伤占主导,bracket 方法完全失效。作者结论是它只适用于缓慢遗忘场景的早期预警。

Original · arXiv cs.LG

Does Learning Order Explain Forgetting? The Lie Bracket as an Early-Warning Signal for Catastrophic Forgetting Without Old Data

A network trained on task A and then task B ends at different weights than one trained on B and then A. To leading order this gap is set by the Lie bracket of the two tasks' gradient fields, and Dherin (2023) conjectured that a non-vanishing bracket may be linked to catastrophic forgetting. I test this conjecture empirically. Across Split MNIST, Split Fashion-MNIST and Split CIFAR-10, the forgetting of A is almost exactly the difference in A's loss between the two training orders (Spearman rho = 0.96-1.00). I then ask whether the bracket can predict that forgetting early in training on B and without any of A's data, using only a small summary stored when A finishes: its gradient and its top 20 Hessian eigenpairs. When the old task is well learned and forgetting is slow (task-incremental), this stored bracket ranks forgetting with rho = 0.70-0.81, against 0.33-0.58 for the penalty of Elastic Weight Consolidation, and matches or exceeds re-evaluating A on its own data. When forgetting is fast (domain-incremental), the stored summary goes stale and the EWC penalty is as good or better. The bracket fails outright when the old task is only half learned; I trace this to a residual gradient on A an order of magnitude larger, which makes first-order (slope) damage matter. The bracket is therefore a useful early-warning signal for the quiet kind of forgetting, under a clear condition.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include continual learning, catastrophic forgetting, Elastic Weight Consolidation, Split CIFAR-10.

Source attribution: curated from arXiv cs.LG (@Ephraim Kofi Ayi Kwapong).