这篇论文用同一个惊喜信号同时解决了持续学习的灾难性遗忘和VLM的元认知问题,效果很漂亮,值得做记忆或VLM的朋友看看。
这篇论文提出,编码器隐空间上的小预测器计算的惊喜信号,既能作为可塑性门控,也能支持元认知。在第一个系统中,基于DINOv2或I-JEPA骨架的非参数情景记忆在持续学习1000类ImageNet时,通过离线复述分别恢复17.7和51.3个百分点的旧类保留率。第二个系统将相同信号用于视觉语言模型,使其在已知概念时断言、陌生时拒绝并请求解释,AUROC达0.966。经过睡眠阶段后,系统能回忆99.2%的教过事实,而基线模型为0%。
Surprise as a Signal for Plasticity and Metacognition
We study a single idea across two settings: that a prediction-error signal, computed by a small predictor over the latent space of a frozen encoder, can serve both as a gate on plasticity and as a substrate for metacognition. In the first system, a non-parametric episodic memory writes a new concept only when this surprise is high, and a periodic offline replay phase consolidates recent traces into a slow linear readout. On a continual stream of 1000 ImageNet classes with a frozen DINOv2 or I-JEPA backbone, the consolidation phase recovers 17.7 points of retention on the oldest classes for DINOv2 and 51.3 points for I-JEPA (single-seed runs), and an ablation shows that replaying only a recent window is worse than no replay at all. In few-shot evaluation the same memory reaches 91.6% on 5-way 1-shot mini-ImageNet, above a task-specific baseline, while a harder 500-way regime exposes the true difficulty. In the second system, the same surprise signal, computed in a shared text-image space, modulates the behaviour of a vision-language model: it answers assertively when a concept is known, hedges when it is partially familiar, and refuses to identify the object and asks for an explanation when it is novel, learning the concept from a single user utterance. The external detector separates known from novel concepts at an AUROC of 0.966 (95% CI +/-0.024), far above the model's own verbalised confidence (0.618), while its token-level confidence sits below chance under greedy decoding; after a sleep phase that empties the fast store, the system recalls 99.2% of fifty taught facts from the consolidated store while a base model recovers none. We report both systems as proof-of-concept, with explicit limitations, and position the second against recent episodic-memory and personalised-VLM work.