研究揭示 Transformer 组合泛化评估中的等价性与模块覆盖问题
Evaluating Compositional Generalization in Transformers: The Role of Composition Equivalence and Module Coverage
这项研究解释了为什么不同论文测组合泛化结论互相矛盾：很多基准里的模块序列其实功能等价，模型是在走捷径。做基准设计或评估模型泛化能力的开发者应该看看，结论会直接影响你怎么切训练测试集。
一篇 arXiv 论文系统评估了 Transformer 的组合泛化能力，指出以往评估结论不一致的原因是训练-测试划分方式随意、覆盖面有限。研究识别出影响性能的两个关键分布因素：组合等价性与模块覆盖率。
实验表明，直接模型（只在最终输出上训练）的良好表现主要来自利用组合等价性，即不同模块序列化简为相同的端到端函数；一旦基准消除这些等价性，性能显著下降，说明这类模型难以泛化到产生新端到端函数的模块组合。同时，模块覆盖偏移越大（同一模块在训练与测试中出现于不同上下文），性能单调下降，直接模型和逐步模型都受影响。
论文以 Llama3-8B 在 LiveCodeBench 代码输出预测任务上的表现为案例，验证了组合等价性的作用。作者据此提出基准设计建议：含组合等价性的基准会高估直接模型的泛化能力，同时包含覆盖违规与等价性的基准会加剧捷径学习。
Evaluating Compositional Generalization in Transformers: The Role of Composition Equivalence and Module Coverage
Compositional generalization is an important form of out-of-distribution generalization, defined as the ability to train on some combinations of modules (e.g., functions such as sort, filter) and then generalize to unseen combinations. Despite substantial empirical work evaluating compositional generalization in transformers, findings have been inconsistent because evaluations were often conducted on a limited set of train-test splits, defined in an ad hoc manner. Through a systematic evaluation, we identify two key aspects of the train-test distribution that strongly affect performance: composition equivalence and module coverage. We show that the apparent performance of direct models (trained only on final outputs) results primarily from exploiting composition equivalences, meaning different sequences of modules that reduce to identical end-to-end functions. When benchmarks eliminate these equivalences, performance drops significantly, showing their inability to generalize to compositions of known modules that produce novel end-to-end functions. We further demonstrate that performance degrades monotonically as module coverage shifts increase (when modules appear in different contexts between train and test), affecting both direct and step-by-step models (trained on intermediate and final outputs). These findings have concrete implications for benchmark design: benchmarks that include compositional equivalences can lead to significant overestimation of the generalization abilities of direct models, and benchmarks that include both module coverage violations and compositional equivalences can exacerbate shortcut learning. As a case study, we demonstrate the role of compositional equivalence on the generalization performance of the pre-trained model Llama3-8B on the code-output prediction task in the coding benchmark LiveCodeBench.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Transformer, 组合泛化, Llama3-8B, LiveCodeBench.
Source attribution: curated from arXiv cs.LG (@Purva Pruthi, Andrew Yuan, Alexander D'Amour, David Jensen).
