Diversify Diffusion with Temperature Sampling and Variance-Corrective Time Shifting 论文

Diversify Diffusion with Temperature Sampling and Variance-Corrective Time Shifting

精选理由

想在不重训模型的情况下让扩散模型生成更多样化的结果?这篇论文给出了一个简单有效的技巧:温度采样配合时间偏移,在DiT、Stable Diffusion上都有效果。

AI 摘要

该论文提出一种无需重新训练的推理方法,通过温度采样(γ<1)和方差校正时间偏移来提升扩散模型的输出多样性。核心思想是:对分数进行γ缩放虽能重加权模式,但会膨胀每模式的方差;而在偏移的时间步查询网络并缩放分数,可抵消方差膨胀。在DiT、Stable Diffusion和Motion Diffusion模型上验证了该方法能一致提升多样性,同时保持样本质量和条件保真度。论文还发现时间干预时机可实现粗到细控制:高噪声阶段驱动跨模式的组合多样性,低噪声阶段驱动固定组合下的局部外观变化。

原文 · arXiv cs.LG

Diversify Diffusion with Temperature Sampling and Variance-Corrective Time Shifting

Diffusion models faithfully reproduce their training distribution, but also inherit its imbalances and leave rare or under-represented modes hard to reach. A natural inference-time remedy is to sample from the high-temperature target $p^{(γ)}_0(x) \propto p_0(x)^γ$ for $0 < γ< 1$, which flattens dominant modes and lifts rare ones. However, naive score scaling while correctly reweighting modes also inflates the per-mode variance, breaking the reverse diffusion process and degrading sample quality. We introduce variance-corrective time shifting, a training-free fix that queries the network at a shifted timestep and scales the resulting score by $γ$, canceling the variance inflation while preserving the mode reweighting. The correction turns simple temperature sampling into a practical diversity knob for pretrained diffusion and flow-matching backbones with no retraining, and we demonstrate consistent gains at minimal cost to sample quality and condition fidelity across DiT, Stable Diffusion and Motion Diffusion models. We further show that the timing of the temperature intervention enables coarse-to-fine control: high-noise stages drive compositional diversity across modes, while low-noise stages drive local appearance variation under a fixed composition.