Complex KDA：增强 Kimi Delta Attention 表达力
Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
欧洲团队改造了 Kimi 的 KDA 注意力，新模型 CKDA 状态追踪更强，代码模型开源，可以上手试试。
OpenEuroLLM 团队提出 Complex KDA（CKDA），让 Kimi Delta Attention（KDA）借助逐通道门控提供的第二次反射实现二维旋转，不增加更新的秩和计算成本。该方法需要同时扩展两处参数范围：门控放宽到 [-1,1]，delta-rule 系数 β 放宽到 [0,2]。理论上，单个 CKDA 层可追踪所有同构于 SO(3) 子群的有限群，状态追踪结果比其他对角加秩一线性 RNN 少用一层。实验上，CKDA 在 S₃、S₄ 和周期音频续写任务上取得所测 KDA 参数范围中最好的长度外推，语言建模成绩超过 Transformer 和其他线性 RNN。代码和模型已在 GitHub 与 Hugging Face 上开源。
Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
Linear RNNs based on the delta-rule enable efficient sequence modeling, but their linear updates with a low-rank correction constrain their expressivity. Prior work has shown that composing two delta-rule transitions in a single recurrent update can model a 2D rotation, but this increases the rank and the cost of the updates compared to a single transition. We show that Kimi Delta Attention (KDA) can realize 2D rotations by combining a single delta-rule transformation with a second reflection supplied by its channel-wise gate. This requires extending the parameter ranges of KDA by combining two existing range extensions: allowing gates in $[-1,1]$ and the delta-rule coefficient $β$ in $[0,2]$. We call the resulting model Complex KDA (CKDA). It preserves KDA's stability and efficiency, with transitions that remain diagonal-plus-rank-one and non-expansive, while reaching the state-tracking expressivity of DeltaProduct$_2$. We characterize the expressivity of CKDA and prove that every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition matrix. A single CKDA layer can track every finite group isomorphic to a subgroup of $\mathrm{SO}(3)$, and many state-tracking results use one fewer layer for CKDA compared to other diagonal-plus-rank-one Linear RNNs. Empirically, combining both extensions yields the strongest length extrapolation among tested KDA range settings on $S_3$, $S_4$, and periodic audio continuation. In language modeling, CKDA outperforms Transformers and other linear RNNs, obtains similar results to a KDA baseline, and shows promising scaling behavior. Our code is open source at https://github.com/OpenEuroLLM/ComplexKDA and our models are available at https://huggingface.co/collections/openeurollm/complexkda.