这篇论文用范畴论统一了注意力、扩散和自条件化，做 Transformer 架构研究的开发者会看到新的理论视角；predict-detach 机制带来的收益比改邻域更大，值得关注。
本文提出 Kan Extension Transformers (KETs)，一个基于范畴论的统一框架，将多种 Transformer 变体（标准注意力、几何 Transformer、扩散模型）视为加权结构化扩展算子的特例。KET 将注意力推广到高阶单纯形邻域，并揭示了与扩散式补全的桥梁。当扩展算子作用于分离的预测载体而非教师强制隐状态时，形成一种有效的自条件化机制，在不泄露未来 token 的情况下暴露非因果结构。在 Penn Treebank、WikiText-2 和 WikiText-103 上的 12 种 Transformer 变体实验中，严格因果设置下二次 KET 在 WikiText-2 和 WikiText-103 上表现最强；但最大收益来自 predict-detach 机制而非邻域族变化。
Kan Extension Transformers: A Categorical Unification of Attention, Diffusion, and Predict-Detach Self-Conditioning
We propose Kan Extension Transformers (KETs) as a unifying categorical framework for a diverse group of Transformer implementations. The core claim is that a Transformer layer can be viewed as a weighted structured extension operator: standard attention is the singleton-neighborhood case, Geometric Transformer style incidence mixing is a sparse edge-restricted case, and KET is the higher-order simplicial case. This lens also clarifies a bridge to diffusion-style completion. When the extension operator acts on detached predictive carriers instead of teacher-forced hidden states, it becomes a valid self-conditioning mechanism that exposes noncausal structure without leaking gold future tokens. We include a comprehensive experimental validation of 12 different Transformer implementations varying across strict-causal and predict-detach regimes on Penn Treebank, WikiText-2, and WikiText-103. In the strict-causal setting, quadratic KET is the strongest model among the compared causal architectures on WikiText-2 and WikiText-103. Across all datasets, however, the largest gains come from the predict-detach regime rather than from changing the neighborhood family alone.