论文提出 Dynamic Wave 混合序列混合器，效率取决于实现而非架构
Attention, Dynamic Wave and Hybrid Sequence Mixers in Language Models
有人做了个注意力替代方案 Dynamic Wave，但真正有意思的是发现：效率优势换个注意力内核就没了，循环层省不省全看实现。
论文形式化了基于输入相关指数衰减与余弦相位调制的 Dynamic Wave 循环层，在 TinyStories 上比较了三种 10.94M 参数配置。Hybrid 配置困惑度最优，为 5.95，优于 Transformer 的 6.93 和 Pure Wave 的 7.26。效率对比显示，对未融合的注意力基线，Pure Wave 在 4096 token 上下文下峰值显存减少 78.7%，但对融合 SDPA 内核则无优势。将逐元素运算移出循环后该层提速 2.53 倍，与注意力的延迟差距从 2.48 倍缩小到 1.12 倍。论文还发现混合架构继承的是更昂贵混合器的成本，其峰值显存与 Pure Transformer 差距在 0.1% 以内。
Attention, Dynamic Wave and Hybrid Sequence Mixers in Language Models
Large Language Models (LLMs) rely on Multi-Head Attention (MHA) for sequence modeling, but scaling to long contexts is constrained by the quadratic cost of attention, motivating sub-quadratic recurrent alternatives. This paper formalizes a Dynamic Wave recurrent layer, based on input-dependent exponential damping and cosine phase modulation, and evaluates three 10.94M-parameter configurations (Pure Transformer, Pure Wave, and Hybrid Wave-Attention) on TinyStories under perplexity, latency, throughput and peak VRAM profiling up to 4096 tokens. The Hybrid configuration reaches the best perplexity, 5.95 on the full held-out split against 6.93 for the Transformer and 7.26 for Pure Wave. The efficiency picture, however, is governed by implementation rather than by architecture. Against an unfused attention baseline, Pure Wave uses 78.7% less peak VRAM at a 4096-token context (1404.3 MB vs. 6601.6 MB); against a fused scaled-dot-product attention kernel the same comparison yields no advantage at all (1404.3 MB vs. 1403.6 MB), and in training mode Pure Wave is 8.3% more expensive than the Transformer. Likewise, hoisting elementwise terms out of the recurrence loop speeds the layer up 2.53-fold with no custom kernel, reducing its latency gap to attention from 2.48-fold to 1.12-fold. A hybrid stack is found to inherit the cost of its more expensive mixer rather than the advantage of its cheaper one, its peak VRAM matching the Pure Transformer to within 0.1%. A closed-form memory model reproduces these measurements and predicts the context length beyond which attention overtakes the vocabulary projection. The findings indicate that reported efficiency gains for recurrent attention replacements must be stated relative to a named attention backend and a named implementation, and that hybridization is best justified on quality grounds.