这个280参数的变换器能完美处理任意长度布尔表达式，比普通模型更擅长深度泛化。
研究人员提出了一种仅含280个可学习参数的变换器参数化方法，能够学习和评估任意深度或长度的问题。该模型通过位置编码跟踪电路中每个门的深度，使用掩码硬注意力识别可求值子表达式。在布尔代数任务中实现了完美的长度泛化，准确率达100%。模型在深度1和深度2的浅层问题实例上训练后，能够完美推广到其他长度泛化基准测试。
Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers
Learning generalizable algorithmic computations remains a challenge for neural networks, as reflected in persistent failures on compositional and length generalization benchmarks. We present a provably correct, transformer parameterization (with only 280 learnable parameters for Boolean algebra tasks) capable of learning and evaluating problems of any depth or length. We assume inputs are fully parenthesized, well-formed expressions. Our approach conceptualizes algorithmic tasks as circuit models embedded in transformers, enabling depth-1 circuit reduction in a single forward pass. To achieve depth generalization, we introduce a positional encoding that tracks each gate's depth within the circuit, enabling the model to identify evaluable subexpressions at each iteration via masked hard attention, with $O(n)$ per-iteration complexity via linear attention. Combined with an autonomous halting criterion, the model terminates after $d$ iterations for problems of depth $d$, yielding $O(n \cdot d)$ total complexity. We show that training on shallow problem instances (depth 1 and depth 2) effectively recovers interpretable parameters that {\em snap} into place, resulting in exact length generalization. Though we establish that our construction provably evaluates Boolean expressions -- a universal symbolic computation -- of arbitrary length perfectly, in other experiments we also demonstrate that our transformer variant can learn and generalize perfectly (100% accuracy) on other common length generalization benchmarks, including modular arithmetic and ListOps.