扩散模型中的特征选择性模型崩溃研究
Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Training
这篇论文揭示了扩散模型在有限预算训练下的特征选择性崩溃机制，对理解模型长期训练稳定性很有价值。
该研究探讨了扩散模型在有限预算训练下的模型崩溃现象。实验在2D螺旋数据集和MNIST、Fashion-MNIST、CIFAR-10图像基准上进行。研究发现，替换协议会导致数据集快速退化，而固定预算协议只会部分退化，保留某些特征。多代参数动力学的线性响应模型分析表明，两种协议存在差异：某些特征在几代内会脆弱丢失，而固定预算协议下某些特征能保持 practically 无限时间。
Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Training
Model collapse arises when generative models are trained on synthetic data produced by earlier models. The phenomenon has attracted considerable attention because of its societal and technical implications. However, previous studies have reached seemingly contradictory conclusions: replacing real data with synthetic data causes collapse (Shumailov et al.), yet accumulating real data alongside synthetic data can prevent it. For diffusion models, we study an intermediate regime typical of finite-budget pipelines: all past datasets and the real data are kept, but each new model is trained on a fixed-size sample from this growing pool, so the real fraction vanishes without any data being removed. Experiments on a 2D spiral dataset as well as the image benchmarks (MNIST, Fashion-MNIST, and CIFAR-10) show that replacement protocol degrades dataset rapidly as in the literature, whereas the fixed budget degrades only partially, sparing some features. A linear-response model of the multi-generational parameter dynamics, analyzed by stochastic recursion, confirms that the two protocols differ: some features will be fragile and lost within a few generations for both protocols, while some will be robust and preserved over practically unbounded horizons under the fixed budget protocol.
