SADL 提出供需匹配约束,将 JEPA 表示崩溃率从 50% 降到 0%
SADL: On the Demand-Supply Geometry of Representation Collapse
做自监督或世界模型的人应该看看这篇。JEPA 训练时表示崩溃很常见,SADL 不加参数、不改架构,只用一个供需匹配的约束就把崩溃率从 50% 压到 0%,有效秩提升好几倍。如果你在用 LeWM 或 INTACT 这类框架,可以直接对照复现。
一篇 arXiv 论文提出 SADL(Supply And Demand Loss),针对 JEPA(联合嵌入预测架构)中的表示崩溃问题。作者发现从初始化开始,编码器表示就被不同输入共享的分量主导,批归一化的 projector 只能在输出端移除该分量,projector 之后的正则化无法修正编码器本身,表示差异持续缩小直至崩溃。
SADL 保留原始训练目标,比较 projector 输出所要求的表示差异与编码器能保留并传递的差异,只在供给不足时对编码器施加约束,不引入辅助网络、额外可训练参数或架构改动,且可与不同 Demand Regularizer 配合使用。
实验数据上,在单任务与四任务联合训练的 LeWorldModel(LeWM)和 INTACT 实验中,SADL 将 10 个随机种子下的崩溃率从 50% 降到 0%;训练 1 个 epoch 后编码器有效秩为基线的 3.4-6.4 倍,潜在预测误差降至基线的 1/20-1/303;配合多个 Demand Regularizer,INTACT 超过 95% 的免搜索控制成功率,在 4 个图像数据集上也一致提升准确率与稳定性。
SADL: On the Demand-Supply Geometry of Representation Collapse
In Joint-Embedding Predictive Architectures (JEPAs), a well-constrained projector output does not by itself prevent representation collapse. We find that, from initialization, the encoder representation is dominated by a component shared by different inputs. A batch-normalized projector removes this component, so regularization after the projector cannot correct it, and the differences between representations shrink until they collapse. We propose SADL (Supply And Demand Loss), which keeps the original training objective and constrains the encoder directly when needed. SADL compares the representation differences that the projector output demands with the differences the encoder can retain and pass on to the projector, and adds a constraint on the encoder only when supply falls short of demand. It works with different Demand Regularizers and introduces no auxiliary networks, additional trainable parameters, or architectural changes. In single-task and jointly trained four-task LeWorldModel (LeWM) and INTACT experiments, SADL lowers the collapse rate over ten random seeds from 50% to 0%. After one epoch of training, encoder effective rank is 3.4-6.4 times that of the corresponding baseline without SADL, and latent prediction error falls to 1/20-1/303 of it. Without additional tuning, SADL stays robust as its loss weight spans orders of magnitude, and with several Demand Regularizers INTACT exceeds 95% mean search-free control success. On four image datasets, SADL also consistently improves accuracy and stability across random seeds. These results show that matching the representation differences a regularizer demands with the differences the encoder can supply leads to faster and more stable representation learning in control and image classification.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include SADL, JEPA, self-supervised learning, world models.
Source attribution: curated from arXiv cs.LG (@Junhan Sun, Yigui Feng, Mingzhe Liu, Haoyan Xu, Lei Ye, Ziqi Han, Hanru Bai, Yanzhao Zhang, Hao Zhao, Guofeng Zhang).