ViTAMINS Vision Transformer Study
ViTAMINS: An Empirical Study of Training Self-Supervised Vision Transformers with Synthetic Hard Negatives
ViTAMINS offers a simpler alternative to generative approaches with better performance and efficiency for vision transformer pretraining.
ViTAMINS introduces synthetic hard negatives for unsupervised vision transformer pretraining. The method improves representation quality by up to +11.3% over baselines on ImageNet and transfer learning tasks. ViT-B outperforms V-JEPA with ViT-L while being more resource efficient. The approach works on image retrieval, copy detection, and image and video segmentation tasks.
ViTAMINS: An Empirical Study of Training Self-Supervised Vision Transformers with Synthetic Hard Negatives
We introduce ViTAMINS, a method that integrates synthetic hard negatives into unsupervised vision transformer pretraining to improve representation quality. Our approach is thoroughly benchmarked on ImageNet and transfer learning, image retrieval, copy detection, and image, video segmentation tasks. Notably, our proposed negatives give rise to emergent properties, where learned representations contain explicit information about the semantic content of an image and serve as excellent classifiers (up to +11.3% over baselines). ViTAMINS achieves these benefits through simple modifications to existing contrastive frameworks and outperforms competing methods while being more resource efficient, e.g., our ViT-B surpasses V-JEPA with ViT-L. Our findings motivate reconsidering contrastive learning as a simpler yet powerful alternative to dominant generative and self-distillation approaches.