Papers

LOOT 架构让掩码扩散语言模型具备自纠错能力

Empowering Masked Diffusion Models to Self-Correct with Leave-One-Out Transformers

Why we picked it

掩码扩散模型一直被人诟病生成后改不了字,这篇论文用 LOOT 架构解决了这个痛点。在 OpenWebText 上推理开销只有之前方法的零头,MAUVE 还更高。做非自回归生成、对推理成本敏感的团队可以认真读一下。

Intel 等机构提出 Leave-One-Out Transformer(LOOT),针对掩码扩散模型(MDM)无法修改已生成 token、训练信号稀疏两大缺陷,在 hollow transformer 基础上重新设计左右上下文融合机制,并推导出更低方差的 MDM NELBO 估计器,使所有 token 都获得监督信号。

Body

推理阶段,LOOT 可以基于上下文独立预测每个位置,从而在生成过程中对 token 进行有效自纠错。基准成绩方面,在 LM1B 数据集上验证困惑度追平最强稠密监督 MDM 基线;在 OpenWebText 上 MAUVE 得分和生成质量-多样性权衡均优于已有 remasking 方法,前向评估次数减少 8 倍,推理 FLOPs 减少 3.5 倍。

对开发者而言,这意味着扩散式并行生成语言模型在效率指标上已具备与自回归方案竞争的资本,self-correction 能力也让非自回归文本生成路线更接近实用。

Original · arXiv cs.LG

Empowering Masked Diffusion Models to Self-Correct with Leave-One-Out Transformers

Masked diffusion models (MDMs) have emerged as a promising paradigm for language modeling, enabling parallel token generation with competitive performance. Yet standard MDMs suffer from two fundamental limitations: (1) inability to revise tokens, and (2) sparse training signal. We revisit hollow transformers, which address both limitations by predicting each token from its surrounding context, independently of the input at that position. We introduce the Leave-One-Out Transformer (LOOT), a more efficient and expressive architecture that advances hollow transformers through two complementary advances. First, we redesign the merging mechanism to enable frequent interaction between left and right contexts, substantially improving predictive performance over prior hollow transformers. Second, we derive a lower-variance estimator of the MDM NELBO that supervises all tokens and improves validation perplexity. Together, these advances yield a competitive standalone diffusion language model capable of effective self-correction during inference. On LM1B, LOOT matches the strongest dense-supervision MDM baseline in validation perplexity. On OpenWebText, LOOT achieves higher MAUVE and better generation quality-diversity tradeoffs than prior remasking methods, using $8\times$ fewer forward evaluations and $3.5\times$ fewer inference FLOPs.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LOOT, MDM, diffusion models, text generation.

Source attribution: curated from arXiv cs.LG (@Sofian Zalouk, Vincent Counathe, Paul Jünger, Daniel Cao, Daniel Zlotnick, Kilian Q. Weinberger, Christopher De Sa).