这篇论文搞清楚了Transformer线性化为什么delta式更厉害，还给出了具体修补方案，在LLaMA和Qwen上跑到了32B，MMLU成绩比之前的后验方法好。
论文揭示了因果自注意力二次复杂度对长上下文推理的瓶颈，在冻结骨干网络下隔离了状态更新设计的效果。研究表明Softmax依赖key相关的秩1正交投影，解释了为何delta式网络优于纯门控累积。作者通过引入sink token、短卷积和固定预算缓存路由等结构干预，减少了近似误差。在LLaMA和Qwen模型上扩展至32B参数，MMLU基准超越了先前后验线性化方法，长上下文检索匹配复杂自适应缓存框架。
The Key to Going Linear: Analysis-Driven Transformer Linearization
The quadratic cost of causal self-attention severely bottlenecks long-context transformer inference. While numerous post hoc linearization pipelines exist, it is difficult to identify which components preserve model quality. This work isolates the effect of state update design in a strict frozen-backbone regime. We show that softmax relies on key-dependent, rank-1 orthogonal projections, elucidating why delta-style networks outperform purely gated accumulation. We identify a potential source of approximation errors and introduce structural interventions, specifically sink tokens, short convolutions, and fixed-budget cache routing, which reduces the remaining gap. We scale this linearization approach across LLaMA and Qwen models up to 32B parameters, outperforming prior post hoc baselines on MMLU and matching the long-context retrieval of complex adaptive-caching frameworks.
- IT之家07-08 05:53原文