CoTA++ 提出免训练方案解决 dMLLM 缓存导致的重复生成问题
CoTA++: Understanding the Repeat Curse in dMLLMs from an Information Flow Perspective
扩散多模态模型用缓存提速后老复读，这篇论文找到三个根因，给了个免训练插件，重复率从 99.6% 压到 17%。
扩散型多模态大语言模型（dMLLM）为加速解码使用跨步骤近似缓存，但会触发 Repeat Curse：响应在相邻位置重复同一个 token。论文从信息流角度定位了三处扰动，包括步内注意力锚定被分散、跨步骤刷新不感知陈旧度、深层上下文熵不收敛。提出的 CoTA++ 框架免训练、即插即用，包含注意力重锚定、解码感知刷新和熵引导投票三个组件。在 3 个 dMLLM 和 2 个缓存后端上，长回复的重复率从最高 99.6% 降到 17.0% 以内，在 9 个基准上恢复了大部分因缓存损失的精度，额外开销低于 15%。
CoTA++: Understanding the Repeat Curse in dMLLMs from an Information Flow Perspective
Diffusion-based multimodal large language models (dMLLMs) decode by iterative denoising with bidirectional attention, which precludes the standard autoregressive key--value cache. Practical deployments therefore rely on approximate caching that reuses intermediate states across denoising steps. Such caching, however, often makes a response repeat one token at adjacent positions in a run that breaks the sentence, a failure we term the \emph{Repeat Curse}. Tracing the context-token information flow reveals three disruptions. Within a step, caching disperses the attention that anchors a position on nearby context tokens. Across steps, similarity-based refresh is not staleness-aware, so some positions are decoded against stale key--value states. Across layers, the deep-layer entropy of the context fails to converge where repetition arises. We accordingly propose CoTA++, a training-free, plug-and-play framework that counteracts each disruption where it originates: Context-Token Attention Re-anchoring re-forms the attention routing against the current context, Decode-Aware Refresh reserves part of the cache's refresh budget for the positions about to be committed, and Context-Token Entropy-Guided Voting discounts the confidence of a position by the deep-layer entropy of its context, which reorders the unmasking. The three components act at non-overlapping stages of a decoding step and compose freely with existing dMLLMs and caches. Across three dMLLMs and two caching backends, CoTA++ reduces the sample repetition rate of long responses from up to 99.6\% to at most 17.0\%, recovers most of the accuracy lost to caching on nine benchmarks, and keeps the speedup of caching at under 15\% overhead, making approximate caching reliable for dMLLMs.
