Yann LeCun 澄清 JEPA 与 JEA 的区别：关键在于预测器
LeCun 亲自科普 JEA 和 JEPA 的区别，一句话说清：区别不在缩写，在要不要预测器，还捎带讲了 30 年谱系
Yann LeCun 发文澄清，JEPA 与 JEA 的核心区别不在缩写本身，而在于是否使用预测器。JEA（联合嵌入架构）直接最小化两个观测 x 和 y 的表示差异并配合防坍缩正则项 R，适用于 x 与 y 是同一场景不同视角、无预测方向的场景，代表方法包括 1989 年 Becker 和 Hinton 的信息最大化模型、NIPS 1993 的 Siamese nets、2020 年 Google 的 SimCLR、2021 年的 Barlow Twins 与 VICReg，以及 FAIR-Paris 的 DINO 系列。JEPA 在此基础上加入预测器 pred(enc(x))，适合非对称预测任务，例如从退化图像恢复、或从当前观测预测未来观测。当预测器加入动作变量 a 时，系统就变成 loss 为 divergence(enc(y), pred(enc(x), a)) + R 的世界模型。
The P in JEPA stands for predictive. The main distinction is not between JEA and JEPA, but between Joint Embedding (predictive or not) and generative architectures. Whether a predictor is necessary depends on the task. Given observations x and y (y being available only during training), a generative architectures predicts y from x. The loss is the divergence between y and its prediction: L = divergence(y, pred(enc(x)) ) A JEA encodes y as well and minimizes the divergence between the representations of x and y, while preventing collapse: L = divergence( enc(y), enc(x) ) + R( enc(y), enc(x) ) Where R is the anti-collapse regularizer, which may be computed over a batch of samples. JEA are used when the prediction has no particular direction, i.e. when x and y are different views of the same scene. The idea for JEA is very old. Becker and Hinton proposed a simple model in 1989 where the anticollapse was based on information maximization. Schmidhuber had a version in 1992 based on redundancy minimization. At NIPS 1993, I proposed Siamese nets, which was a contrastive method to train JEA. Such contrastive methods gained popu.arity after Google's SimCLR paper in 2020. In 2021, Stéphane Deny in my group proposed the Barlow Twins redundancy reduction regularizer, which seemed to work at scale. This was shortly followed by VICReg by Adrien Bardes, Jean Ponce, and me, and last year by SIGReg by Randall Balestriero and me. DINO is a JEA method that comes from a long string of papers from Piotr Bojanowski's group at FAIR-Paris, starting with Noise as Targets (2017), Deep Clustering (2018-2020), and DINO v1-v3 which are contrastive between clusters, not samples. JEPA adds a predictor, which is useful when the prediction is non-symetric. For example, when x is a degraded/corrupted version of y, or when x is a present observation and y a future observation (temporal prediction). The loss is: L = divergence( enc(y), pred(enc(x)) ) + R( enc(y), enc(x) ) All of these models can be augmented by an action variable a, fed to the predictor, encoding an action, intervention, or transformation to produce y from x. This turns the system into a "world model". In a JEPA, the loss becomes L = divergence( enc(y), pred(enc(x), a) ) + R( enc(y), enc(x) ) 💬 8 🔄 21 ❤️ 142 👀 9621 📊 30 ⚡
