论文揭示自适应序列模型中隐藏的未来Token泄漏问题
Beyond Causal Masks: Auditing and Localizing Hidden Future-Token Leakage in Adaptive Sequence Models
训练损失降了一半多但生成质量反而更差，这篇论文告诉你怎么用前缀不变性审计和 LeakScore 抓出这种假进步。
arXiv 论文研究了自适应残差路由解码器中的未来Token泄漏：尽管因果注意力掩码正确，序列级路由摘要仍通过动态路由查询路径将前缀位置暴露给未来Token。一个 24 层异常检查点将验证损失从基线的 3.5958 降到 1.4259，但未通过前缀不变性审计，自由生成时的参考模型困惑度也明显更高。作者用前缀不变性契约、LeakScore（最大前缀输出差异）及逐层诊断定位问题，两个独立训练的 12 层修复版本消除了泄漏，验证损失和困惑度回到对照水平。实验覆盖六个合成实现对、五个模块家族，以及数值精度下的缓存一致性检查。
Beyond Causal Masks: Auditing and Localizing Hidden Future-Token Leakage in Adaptive Sequence Models
Hidden future-token leakage can create false architectural gains: an autoregressive model may appear substantially better under teacher-forced evaluation while its free-running generation deteriorates. We investigate this failure in an adaptive residual-routing decoder whose sequence-level routing summary exposes prefix positions to future tokens despite a correct causal attention mask. An anomalous 24-layer checkpoint reduces validation loss from the baseline's 3.5958 to 1.4259, yet it fails a prefix-invariance audit and has substantially higher reference-model perplexity during generation. Layer-wise diagnostics and implementation inspection trace the violation to a full-sequence summary broadcast through the dynamic routing-query path. Two independently trained 12-layer causal repairs eliminate detectable leakage and return validation loss and reference perplexity to the control scale, removing the large apparent advantage. As diagnostic tools, we use the established prefix-invariance contract, maximum prefix-output discrepancy (LeakScore), complementary behavioral metrics, and layer-wise profiles. Supporting experiments cover six synthetic implementation pairs spanning five module families, clean pretrained controls, controlled fault injection, and cache consistency across numerical precisions. These results illustrate how hidden future-token access can produce misleading claims of architectural progress and why lower teacher-forced loss should be interpreted together with whole-model causal checks and free-running evaluation.
