SMD：纯回归式少步蒸馏框架，4步生成可反超多步教师模型
Simplifying and Improving On-Policy Distillation for Few-Step Diffusion
这篇论文把少步蒸馏简化成纯回归，4步生成就能反超多步教师，还做到了33B参数规模。
论文提出 Simplified Matching Distillation（SMD），将扩散与流模型的少步蒸馏简化为在策略速度回归，以此支撑 4-NFE 生成。方法引入名为 Pivot 的仅训练期残差代理，用来表达教师在每个粗粒度转移内的速度变化，并通过排斥较弱教师预测来避开不良模式。整个框架完全基于回归，不需要在线分数估计或对抗训练，部署时的生成器结构保持不变。论文在图像生成与编辑、3D 生成、视频合成和音视频联合生成上评估，规模扩展到 330 亿参数。在 4-NFE 设置下，SMD 超过多个少步基线，并在若干基准上超越多步教师模型，同时支持跨模型、多教师和因果视频蒸馏。
Simplifying and Improving On-Policy Distillation for Few-Step Diffusion
Few-step distillation reduces the sampling cost of diffusion and flow models, but existing approaches often require auxiliary score models or specialized architectures. We show that simple on-policy velocity regression already provides a strong foundation for 4-NFE generation. To represent teacher velocity variation within each coarse transition, we introduce Pivot, a training-only residual proxy that separates local corrections from the student's interval-mean prediction. We further repel the student from weaker teacher predictions, providing a lightweight mechanism for avoiding undesirable modes. The resulting framework, Simplified Matching Distillation (SMD), remains entirely regression-based, requires neither online score estimation nor adversarial training, and leaves the deployed generator unchanged. We evaluate SMD on image generation and editing, 3D generation, video synthesis, and joint audio--video generation, scaling to 33 billion parameters. At 4-NFE, SMD outperforms strong few-step baselines and surpasses multi-step teachers on several benchmarks. It also supports cross-model, multi-teacher, and causal-video distillation.
