研究揭示 I-JEPA 预测器宽度决定masked latent prediction保留哪些信息
Shared Is Not Enough: Predictor Width and Factor Readability Under Masked Latent Prediction
I-JEPA 表现不如 SimCLR 的原因找到了：预测器太窄，加宽后直接反超，还附了模拟器和模型可以自己跑。
arXiv 论文构造了一个每个图像因子已知的模拟器，用于逐因子测量训练后特征的保留程度。研究用 I-JEPA、SimCLR、BYOL 训练 0.7M 参数的小型编码器并探测共享因子，I-JEPA 的中位数得分约为 SimCLR 的 40%。将 I-JEPA 预测器宽度从编码器的一半加宽到相同宽度后，结果在每个训练种子下反超另外两种方法。模拟器和训练好的模型已开源。
Shared Is Not Enough: Predictor Width and Factor Readability Under Masked Latent Prediction
Masked latent prediction methods such as I-JEPA aim to learn predictable features, but what determines which information they retain is less clear. We built a simulator in which every factor behind an image is known, so a probe can measure how readable each factor remains after training. We trained small encoders (0.7M parameters) with I-JEPA, SimCLR and BYOL on the same pairs of views and probed a factor that both views share. I-JEPA had a predictor with half the encoder's width and depth, the proportions it uses for ViT-B. Its median score was about 40% of SimCLR's. Widening the predictor to the encoder's width reversed this ordering at every training seed, against both other methods. We release the simulator and the trained models.