这篇论文讲了Transformer的理论短板——表达性研究够了，但学不学得会还不知道。他们用C-RASP给出了样本复杂度的初步界限，对想深挖模型理论的人很有用。
这篇论文聚焦Transformer的理论理解，指出已有研究大多分析其表达性，但很少涉及可学习性。受损失景观分析启发，作者初步提出了学习C-RASP构造的样本复杂度边界。该工作为理解Transformer在有限样本下的学习能力提供了理论基础。
From Expressivity to Sample Complexity: Narrow Teachers for Transformers via C-RASP
A theoretical understanding of Transformers is crucial to better understand the capacities and limitations of large language models (LLMs). There is much work analyzing the expressivity of attention-based models. By proposing handcrafted weights or using computational complexity arguments, a large amount of past theoretical works have sought to characterize which tasks are and which are not in the hypothesis class of Transformer models. However, little work investigates the learnability of such solutions. In this work, we make progress towards this goal. Inspired by recent loss landscape analysis work, we propose preliminary sample complexity bounds for learning C-RASP constructions with Transformers.