论文精选73°

块状生成中的信息下限与模型差距研究

Beyond Parallel Blindness: Information Floors and Model Gaps in Block Drafting

精选理由

这篇论文揭示了块状生成器中信息下限与模型差距的关系,解释了为什么当前生成器仍有很大改进空间。

AI 摘要

该研究针对块状生成器(block drafters)提出了一种信息下限(information floor)概念,用于分离两种损失类型。在Qwen3-4B模型上,全并行信息下限最终达到0.286,限制了最佳提案的每槽接受率仅为71%。研究还发现,一个已生成标记可消除86%-100%的信息下限。当前生成器仍远高于其信息下限,最终槽位的模型差距占DFlash拒绝的43%-64%和DSpark的oracle条件拒绝的85%-92%。

原文 · arXiv cs.LG

Beyond Parallel Blindness: Information Floors and Model Gaps in Block Drafting

Block drafters propose several tokens in one forward pass, before earlier target tokens are realised. Their rejection mixes two losses: missing within-block path information and imperfect modelling of observable information. Accepted length cannot distinguish them. We separate the two with an information floor, the minimum expected rejection at a specified conditioning order; rejection above this floor is the model gap. Estimating both from target rollouts across four domains, four open-weight targets, and a frontier API target yields three findings. First, the all-parallel floor reaches $0.286$ at the final slot on Qwen3-4B, limiting even the best proposal to $71\%$ per-slot acceptance. Second, one realised token removes $86$--$100\%$ of this floor, a locality also recovered by an independent mutual-information analysis. Third, current drafters remain far above their floors: the final-slot model gap accounts for $43$--$64\%$ of DFlash rejection and $85$--$92\%$ of DSpark's oracle-conditioned rejection. These findings separate the value of short-range conditioning from proposal quality.