LAW 提出纯 Self-Conditioning 生成模型，扩散语言模型新路线
Loopholing All the Way: Language Modeling with pure Self-Conditioning
一篇扩散语言模型论文，把 Self-Conditioning 从辅助技巧升级成整套生成框架，GSM8K 上 42.2% 对 34.5%，还把 Duo 微调后翻快三倍，做生成模型研究的可以看看。
arXiv 论文提出 LAW（Loopholing All the Way），将传统上被视为辅助技巧的 Self-Conditioning 变成连续生成模型的核心机制，通过跨步传播去噪器预测的 Expected Embedding，最后一次性解码 token。在 Sudoku 任务上，LAW 超越此前的离散与连续扩散模型，包括 Loopholing Discrete Diffusion。在 Maze 任务上，LAW 在 2 到 64 NFE 区间准确率最高。在 TinyGSM 上训练并以 1024 NFE、温度 1 采样后，LAW 在 GSM8K 上解题率 42.2%，高于均匀 Loopholing Discrete Diffusion 的 34.5%。用 LAW 微调预训练扩散模型 Duo 后，准确率从 17.2% 升至 48.1%，在 OpenWebText 上则以 64 NFE 接近自回归模型。
Loopholing All the Way: Language Modeling with pure Self-Conditioning
Recent Diffusion Language Models (DLMs) use Self-Conditioning (SC), or Loopholing, to improve their performance. While SC is traditionally seen as an auxiliary heuristic, its empirical success raises the question: can a continuous generative model be based purely on Self-Conditioning? We answer affirmatively with LAW (``Loopholing All the Way''), which propagates the Expected Embedding under the denoiser's prediction across steps, decoding the tokens once, at the end. On Sudoku, LAW outperforms previous Discrete and Continuous Diffusion Models, including Loopholing Discrete Diffusion. On Maze, LAW is the most accurate from 2 to 64 NFE. After training on TinyGSM and sampling at temperature 1 with 1024 NFE, LAW solves 42.2% of GSM8K problems against 34.5% for uniform Loopholing Discrete Diffusion. Additionally, fine-tuning a pre-trained Uniform Diffusion Model (Duo) with LAW increases its accuracy from 17.2% to 48.1%. Finally, on OpenWebText, LAW pushes the Pareto frontier and approaches Autoregressive Models (ARMs) with 64 NFE. Find our code at https://github.com/ilievanadezhda/law.
