TCDA:为扩散语言模型自适应选择去噪调度,最高省 8 倍算力
One Schedule Does Not Fit All: Adaptive Inference Schedules for Diffusion Language Models
扩散语言模型跑推理时其实不用每次都走完整去噪流程。这篇论文给了一套带统计保证的选择规则,LLaDA-8B 在 WMT14 上能省 8 倍计算。做部署或研究扩散 LM 的工程师可以直接照着实现,论文还提醒 ELF 采样器时间网格是随机的,对比实验前要先固定。
一篇 arXiv 论文提出 Task-Calibrated Denoising Allocation(TCDA),用校准数据加单侧 bootstrap 置信界,为扩散语言模型逐任务认证最便宜的去噪调度,部署不可行调度的概率渐近不超过 alpha。
候选方案包括早期前缀解码、状态复用续写、缩减时间网格和置信度阈值解码,认证失败则回退完整调度。实测中,ELF-B 在 XSum 上把去噪器评估次数降到原来的 1/3.
4,在 WMT14 上降到 1/2.0;
LLaDA-8B 在 WMT14 上降到 1/8.0,在 GSM8K 上降到 1/4.
1;ELF-M 与 ELF-L 在 OpenWebText 上均降到 1/1.
33。在 500 个校准划分上,名义水平 5% 时虚假认证率始终低于 2%,且预先固定的验证协议在新种子上全部复现。
论文还指出 ELF 参考采样器随机抽取时间网格,无条件生成的可复现对比需固定网格;轨迹测量显示已解码 token 的提交程度比剩余潜变量离散化误差更适合做调度选择信号。
One Schedule Does Not Fit All: Adaptive Inference Schedules for Diffusion Language Models
Diffusion language models typically use a fixed denoising schedule across workloads. We show that the cheapest schedule that preserves quality depends on the task, model, and sampler. We propose Task-Calibrated Denoising Allocation (TCDA), a deployment rule with an explicit error guarantee. Given candidate samplers, paired calibration data, a utility tolerance, and guardrails against degeneration, TCDA uses simultaneous one-sided bootstrap bounds to certify which candidates meet both the utility and guardrail criteria. It deploys the cheapest certified candidate or falls back to the full schedule. The probability of deploying an infeasible schedule is asymptotically at most alpha. Candidates include early prefix decoding, state-reusing continuation, reduced time grids, and confidence-threshold decoding. Across seven datasets and two model families, TCDA certifies reductions in denoiser evaluations by factors of 3.4 on XSum and 2.0 on WMT14 for ELF-B; 8.0 on WMT14 and 4.1 on GSM8K for LLaDA-8B; and 1.33 on OpenWebText for both ELF-M and ELF-L. A validation protocol fixed before generation confirmed every selection on fresh seeds. Across 500 calibration splits, false certification remained below 2% in every setting at a nominal level of 5%. Certified schedules vary across tasks and model scales. Trajectory measurements suggest that when decoded tokens commit is more informative for schedule selection than the remaining latent discretization error. We also find that the ELF reference sampler draws its time grid at random, so reproducible schedule comparisons for unconditional generation require fixing the grid.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include TCDA, LLaDA, ELF, diffusion language models.
Source attribution: curated from arXiv cs.LG (@Rabimba Karanjai, Linsey Pang, Hemanth Hegadehalli Madhavarao, Richard Williamson, Uma Kona, Prakhar Mehrotra).