NVIDIA 新线性注意力，解耦擦写门
NVIDIA 发布 Gated DeltaNet-2，一种线性注意力层，将 Delta 规则中的擦除和写入操作解耦为通道级擦除门 b_t 和写入门 w_t。在 1.3B 参数、100B FineWeb-Edu 令牌训练下，它在语言建模、常识推理和长上下文检索任务上超越 Mamba-2、Gated DeltaNet、KDA 和 Mamba-3。最大提升出现在 RULER S-NIAH 和多键针检索基准上。
NVIDIA AI Releases Gated DeltaNet-2: A Linear Attention Layer That Decouples Erase and Write in the Delta Rule
Linear attention squeezes the unbounded KV cache into a fixed-size recurrent state, but editing that memory without scrambling existing associations is hard. Prior delta-rule models like Gated DeltaNet and KDA use one scalar gate to control both erasing old content and writing new content. NVIDIA's Gated DeltaNet-2 decouples these into a channel-wise erase gate b_t on the key axis and a channel-wise write gate w_t on the value axis. At 1.3B parameters trained on 100B FineWeb-Edu tokens, it outperforms Mamba-2, Gated DeltaNet, KDA, and Mamba-3 across language modeling, commonsense reasoning, and long-context retrieval — with the largest gains on RULER S-NIAH and multi-key needle retrieval. The post NVIDIA AI Releases Gated DeltaNet-2: A Linear Attention Layer That Decouples Erase and Write in the Delta Rule appeared first on MarkTechPost .