MPPA：免训练方法将扩散语言模型 LLaDA-8B 上下文扩展到 2 倍窗口
Mask-Pattern Position Augmentation: Training-Free Long-Context Extrapolation for Diffusion LLMs
扩散语言模型长上下文外推一直是难点，这篇论文没做任何训练就把 LLaDA-8B 的 8192 token 困惑度从 19.51 降到 3.12，比窗口内还低。思路是利用解码过程中的掩码密度给 RoPE 加动态偏移，和自回归那边静态的 NTK 缩放完全不同。做 dLLM 推理或长文本任务的工程师可以直接看方法细节。
一篇 arXiv 论文提出 Mask-Pattern Position Augmentation（MPPA），针对扩散大语言模型（dLLM）的上下文外推问题，无需任何训练。dLLM 通过双向注意力迭代解码掩码 token，但 RoPE 仍将其限制在预训练窗口内，现有方法 LongLLaDA 直接套用自回归模型的 NTK 静态 RoPE 缩放，忽略了 dLLM 特有的动态掩码信号。
MPPA 的做法分两步：先对位置索引做线性插值压缩进预训练窗口，再根据累计掩码密度为每个位置、每个去噪步加入动态偏移量。该偏移恢复了被插值压缩掉的旋转相位分辨率，且随着 token 被确定而逐渐减小，使后期去噪步接近干净的插值几何。
在 LLaDA-8B-Instruct 上，模型预训练窗口为 4096 token，MPPA 将 8192 token 处的困惑度从标准解码的 19.51 降至 3.12，低于窗口内水平，同时 passkey 检索通过率保持 100%，needle-in-a-haystack 准确率不变。结果表明基于掩码模式的位置工程是扩散模型原生的长上下文扩展路径。
Mask-Pattern Position Augmentation: Training-Free Long-Context Extrapolation for Diffusion LLMs
Diffusion large language models (dLLMs) generate by iteratively unmasking tokens with bidirectional attention, but rotary position embeddings (RoPE) still confine them to a fixed pretraining window. Existing training-free extrapolation, notably LongLLaDA, copies NTK-based RoPE scaling from autoregressive (AR) models and applies one static, length-only transform at every position and denoising step. That transplant ignores a signal unique to dLLMs: the evolving mask pattern of the partially decoded sequence. We propose Mask-Pattern Position Augmentation (MPPA). MPPA linearly interpolates indices into the pretrained window, then adds a per-position, per-step offset from cumulative mask density. The offset restores rotary phase resolution that interpolation collapses, and it shrinks as tokens are committed, so late steps see near-clean interpolated geometry. On LLaDA-8B-Instruct at twice its 4096-token window, MPPA reduces perplexity from 19.51 under standard decoding to 3.12 at 8192 tokens, below the in-window figure, with 100% passkey retrieval and unchanged needle-in-a-haystack accuracy. These preliminary $2\times$ results motivate mask-conditioned position engineering as a diffusion-native path to long-context generation.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LLaDA, dLLM, RoPE, 长上下文.
Source attribution: curated from arXiv cs.LG (@Gaurav Sarkar, Syed Affan Daimi).
