GTFormer 用图网络替代自注意力，实现 8192 倍长度外推
GTFormer: A Syntax-Free Graph Network Alternative to Transformers for 8192$\times$ Length Extrapolation in Linear Time
用图网络替掉自注意力，训练只到 512 长度就能零样本跑 400 万 token，线性内存还不爆显存，思路挺新鲜。
论文提出 GTFormer，一种无语法依赖的分层图神经网络，用平衡二叉归约树、边界桥接边和随机扩展器捷径替代扁平自注意力。在八个长上下文代码、日志和多跳检索基准上，模型仅在 N≤512 的序列上训练，即可零样本外推到 N=4,194,304，即 8192 倍于训练长度。推理在单张消费级 GPU 上以线性 O(N) 内存完成，而标准 Transformer 和 RoPE Transformer 在该长度下出现注意力坍缩或显存耗尽。
GTFormer: A Syntax-Free Graph Network Alternative to Transformers for 8192$\times$ Length Extrapolation in Linear Time
Self-attention mechanisms have established the Transformer as the dominant backbone across natural language processing, source code intelligence, and automated theorem proving. However, all-to-all pairwise token comparison enforces a quadratic $\mathcal{O}(N^2)$ compute and memory barrier alongside severe softmax attention dilution as sequence length $N$ scales beyond the training horizon. Inspired by recent advances in algebraic graph learning (\textbf{WPNet}), this work introduces \textbf{GTFormer}, a syntax-free hierarchical Graph Neural Network (GNN) that replaces flat self-attention for general long-context sequence modeling. GTFormer organizes raw token streams into a bounded-degree hierarchy combining a balanced binary reduction tree ($\lceil \log_2 N \rceil$ depth), horizontal boundary-bridging shift edges, random expander shortcuts, multi-channel convex highway gating, and a top-down broadcast pass. Across eight long-context code, log, and multi-hop retrieval benchmarks trained exclusively on sequences of length $N \le 512$, GTFormer extrapolates zero-shot to $N = 4{,}194{,}304$ ($8192\times$) on a single commodity GPU in linear $\mathcal{O}(N)$ memory, maintaining near-perfect accuracy where standard and RoPE Transformers suffer from out-of-distribution attention collapse and out-of-memory exhaustion.