这篇论文证明了MIM比CL更抗非IID数据，还发现联邦学习在鲁棒性上不比分散式学习差。搞分布式自监督的开发者别错过。
该论文理论分析了分布式自监督学习（D-SSL）框架在非独立同分布（non-IID）数据下的鲁棒性。研究发现，基于掩码图像建模（MIM）的预训练比对比学习（CL）对异构数据更鲁棒。鲁棒性随平均网络连接性增加而提高，表明联邦学习（FL）不弱于分散式学习（DecL）。作者提出MAR损失，通过局部到全局对齐正则化改进MIM目标，实验验证了理论并证明其有效性。
Understanding the Robustness of Distributed Self-Supervised Learning Frameworks Against Non-IID Data
Recent research has introduced distributed self-supervised learning (D-SSL) approaches to leverage vast amounts of unlabeled decentralized data. However, D-SSL faces the critical challenge of data heterogeneity, and there is limited theoretical understanding of how different D-SSL frameworks respond to this challenge. To fill this gap, we present a rigorous theoretical analysis of the robustness of D-SSL frameworks under non-IID (non-independent and identically distributed) settings. Our results show that pre-training with Masked Image Modeling (MIM) is inherently more robust to heterogeneous data than Contrastive Learning (CL), and that the robustness of decentralized SSL increases with average network connectivity, implying that federated learning (FL) is no less robust than decentralized learning (DecL). These findings provide a solid theoretical foundation for guiding the design of future D-SSL algorithms. To further illustrate the practical implications of our theory, we introduce MAR loss, a refinement of the MIM objective with local-to-global alignment regularization. Extensive experiments across model architectures and distributed settings validate our theoretical insights, and additionally confirm the effectiveness of MAR loss as an application of our analysis.