DiffPT：用扩散模型预训练分类器，小样本场景更有效
Data-Efficient Classifier Pretraining Through Diffusion Models
这篇论文教你在数据很少时怎么给分类器做预训练：DiffPT 用扩散模型，医学影像和时间序列上比自监督方法更扛用。
arXiv 论文提出 DiffPT，一种用扩散模型对分类器骨干做预训练的方法，面向领域训练数据有限的场景。实验覆盖时间序列、标准自然图像和医学影像三类数据。在 CIFAR-10 和 Tiny ImageNet 的低数据量设定下，对比式预训练表现更好；但在医学影像和时间序列任务上，DiffPT 在多个低数据量区间超过自监督基线。方法动机来自 AlexNet 之前基于重建的自动编码器预训练思路。
Data-Efficient Classifier Pretraining Through Diffusion Models
Machine learning with limited data is susceptible to overfitting, especially with modern deep neural networks. Prior to the introduction of AlexNet, reconstruction-based pretraining with auto-encoder and denoising auto-encoder were found to be beneficial, partially due to the limited size of the training sets at the time. Motivated by these traditional approaches, we propose DiffPT, a simple diffusion modeling approach to pretrain a given classifier backbone for limited domain-specific training data. We experiment on time-series data, standard natural-image benchmarks, and medical imaging datasets, and contrast the diffusion approach to a multitude of modern and traditional un/self-supervised methods. While contrastive pretraining performs better on CIFAR-10 and Tiny ImageNet in low-data regimes, DiffPT is particularly effective on medical imaging and time-series tasks, where it outperforms self-supervised baselines across a range of low-data regimes. Our results suggest that DiffPT is a simple and effective approach for data-efficient classifier pretraining on different modalities.
