arXiv 论文提出生成式表示学习方法， tokenizer 可扩展且计算更优
Generative Representation Learning for Scalable Latent Tokenization
一篇挺硬核的表示学习论文：作者把 tokenizer 和生成模型放一起算 flop 效率，结果比 DINO 和 LeJEPA 更抗塌缩，还给出了 scaling 规律。
一篇 arXiv 论文提出一种自监督表示学习方法，用作生成式建模的 tokenizer，通过对数据的逐点互信息做分解来监督表示损失。论文从理论和实验上说明该方法能避免 DINO 和 LeJEPA 等常用方法出现的维度塌缩和特征抑制。配合 JiT-L 模型，在像素空间生成上达到 FID-50K 2.18，相比直接扩散生成是计算最优的改进。在 ImageNet 上该方法比 DINOv2 少用 7 倍 FLOPs，在文生图数据集 GPIC 上比 WebSSL 少用 8 倍 FLOPs。
Generative Representation Learning for Scalable Latent Tokenization
We introduce an approach for self-supervised representation learning designed to be used as a tokenizer for generative modeling. Our objective supervises a representation loss by factorizing the pointwise mutual information of the data, which we estimate by taking information differences between conditional predictions from a learned generative model. We justify our algorithm theoretically and empirically, demonstrating that our generative approach to representation learning avoids dimensional collapse and feature suppression in cases where commonly used methods like DINO and LeJEPA fail. Moreover, we show our representation learning technique is scalable; expending more compute leads to predictably better latent tokenization. For the first time, we perform a careful analysis of the combined flop efficiency of tokenizer and generative model and demonstrate that our approach is a compute-optimal improvement over direct diffusion for pixel-space generation, obtaining an FID-50K of 2.18 with JiT-L. Applied to ImageNet and the text-to-image dataset GPIC, we show tokenizer encoder scaling laws for diffusability, where extrapolating our approach achieves representations in 7x fewer FLOPs than DINOv2 on ImageNet and 8x fewer FLOPs than WebSSL on GPIC.