智谱发布混合注意力扩散语言模型dQwen3.5系列
dQwen3.5: Hybrid-Attention Diffusion Language Models
智谱新出的dQwen3.5系列模型，用混合注意力架构改造自Qwen3.5，在训练效率上比全注意力架构有优势。
智谱研究团队将Qwen3.5模型改造为扩散语言模型dQwen3.5，在0.8B、2B、4B和9B规模下测试，发现混合注意力架构比全注意力架构更高效，训练损失达到相同水平时消耗约一半的token数量。
dQwen3.5: Hybrid-Attention Diffusion Language Models
Adapting a pretrained autoregressive (AR) model is a cost-efficient route to a diffusion language model (DLM). While nearly all such adaptations start from a full-attention transformer, AR modeling has shifted toward hybrid architectures that interleave attention and RNN layers. This creates an obstacle for adaptation: unlike attention, RNNs are structurally causal and nontrivial to bidirectionalize. Despite this mismatch, we investigate whether such backbones can become effective DLMs by adapting Qwen3.5 at 0.8B, 2B, 4B, and 9B scales, yielding the dQwen3.5 family. We find that hybrid backbones can be efficient starting points for adaptation: against a full-attention control, the hybrid reaches a given training loss in about half the tokens. Across scales, dQwen3.5 resembles full-attention DLMs in any-order decoding behavior and performs strongly under parallel decoding.
