研究S-JEPA编码器中非极大概率与GMM成分映射的作用

Does Mapping Non-Maximal Probabilities to GMM Components Matter for S-JEPA Encoder Representations?

精选理由

朋友间可以说:你关注S-JEPA的话,这篇讲它编码器里概率和GMM成分映射的研究值得看看,用不同方法对比后发现映射很重要,和普通做法比效果不同哦。

AI 摘要

在S-JEPA编码器研究中,以GMM软后验为基础,测试非极大概率分配方式对表现的影响;采用REAL SOFT、FIXED-RANDPERM、UNIFORM-TAIL三种对照方法,对比不同映射下的实验结果;三组独立实验显示,非极大概率到GMM成分的映射确实影响编码器学习效果。

原文 · arXiv cs.LG

Does Mapping Non-Maximal Probabilities to GMM Components Matter for S-JEPA Encoder Representations?

S-JEPA uses soft Gaussian mixture model (GMM) posteriors instead of hard cluster labels to preserve uncertainty. It remains unclear whether the probability values alone are sufficient, or whether it also matters which GMM components receive the non-maximal probabilities. We test this with two matched controls. FIXED-RANDPERM keeps the top-1 component and probability together with the multiset of non-maximal probability values, but reassigns those non-maximal values using a mapping fixed for each physical frame. UNIFORM-TAIL keeps the top-1 component, its probability, and total non-maximal mass but distributes that mass uniformly. Across three independent seeds, REAL SOFT outperforms both controls on two frozen Encoder readouts. It provides better recovery of the original GMM tail and greater accessibility of spectral dynamics over short time scales after controlling for the complete spectrum of the current frame. In two exposure experiments, both readouts improved overall as more frames retained the original mapping. We also descriptively follow one Phase 2 trajectory after the switch to the online GMM. These results show that the numerical probability structure of the soft target does not fully determine the learned Encoder representation. The mapping of non-maximal probabilities to GMM components also matters.