这篇论文给Transformer的提示调优、微调等技术找到了数学理论，解释了为什么注意力机制能压缩信息。
论文提出一个基于分布回归的Transformer学习框架，将两阶段采样过程与自然语言处理关联。定义了注意力算子，证明Transformer可无损压缩分布为函数表示。相比卷积神经网络和全连接网络，Transformer在更复杂结构的功能学习上表现更强。该框架还为大语言模型中的提示调优、参数高效微调、高效缩放等技术提供理论洞见。
Generalization Analysis of Transformers in Distribution Regression
In recent years, models based on the Transformer architecture have seen widespread applications and have become one of the core tools in the field of deep learning. Numerous successful techniques, such as parameter-efficient fine-tuning and efficient scaling, have been proposed surrounding their applications to further enhance performance. However, the success of these strategies has always lacked the support of rigorous mathematical theory. To study the underlying mechanisms behind Transformers and related techniques, we first propose a Transformer learning framework motivated by distribution regression, with distributions being inputs, connect a two-stage sampling process with natural language processing, and present a mathematical formulation of the attention mechanism called attention operator. We demonstrate that by the attention operator, Transformers can compress distributions into function representations without loss of information. Moreover, with the advantages of our novel attention operator, Transformers exhibit a stronger capability to learn functionals with more complex structures than convolutional neural networks and fully connected networks. Finally, we obtain a generalization bound within the distribution regression framework. Through the aforementioned theoretical results, we further discuss some successful techniques emerging with large language models (LLMs), such as prompt tuning, parameter-efficient fine-tuning, and efficient scaling. We also provide theoretical insights behind these techniques within our novel analysis framework.