Library of Layers：推理时动态调整 Transformer 层顺序的新架构
Library of Layers (LoL): Adaptively Ordering Transformer Layers at Runtime
arXiv 上这篇论文挺有意思：LoL 让模型推理时自己决定 transformer 层的执行顺序，把稠密、循环、MoE 三种架构统一到一套框架里，还能通过干预路由切换解题策略。
arXiv 论文提出 Library of Layers（LoL），在推理阶段动态排列 transformer 层的执行顺序。该架构仅有一个超参数，即全局 transformer 块库的大小，可将稠密、深度循环和 MoE 架构的参数效率与 FLOPs 效率权衡统一到同一框架。在文本预训练上，LoL 性能与三类基线持平；在合成 sudoku 和 maze 数据上，模型对不同解法使用不同的层顺序，且可通过干预路由强制切换策略。多语言实验显示层顺序随语言和文字系统变化，安全数据上的路由探针能以高 AUROC 区分安全与不安全文本。
Library of Layers (LoL): Adaptively Ordering Transformer Layers at Runtime
Computer programmers chain together the same reusable computational primitives, for example, conditional statements or loops, but in different orders to implement different algorithms. In contrast, existing language model architectures always execute their layers in the same order. We introduce Library of Layers (LoL), an architecture that dynamically orders its layers during inference to solve the problem at hand. LoL has a single hyperparameter, the size of its global library of transformer blocks, that allows it to transition seamlessly between parameter reuse and sparsity, generalizing the parameter-efficiency and FLOPs-efficiency tradeoffs of dense, depth-recurrent, and mixture of experts architectures in one unified family. We show that LoL achieves competitive performance with all three above baselines when pretrained on text data. On synthetic sudoku and maze data, we observe that the model employs distinct layer orderings to implement different solution strategies, and we can intervene on routing to force a model to apply a specific solution strategy to a problem instance. In a multilingual natural language setting, we find that LoL chooses different layer order patterns based on language and script. Similarly, we train LoL on safety data and fit probes to its router outputs, finding that these probes distinguish safe from unsafe text with high AUROC. LoL provides a path toward a future in which models dynamically compose their own computational graph, enabling more efficient and controllable reasoning.