做NLP模型压缩或可解释性研究的开发者，可以关注这种用语法知识替代暴力稀疏化的思路——既省算力又不掉点，值得在长文本任务上试试。
该论文提出一种基于语法角色（词性标注）的稀疏注意力机制，通过动态生成注意力掩码，只允许语法相关的词对进行交互，从而降低Transformer自注意力的计算复杂度。实验在SST-2情感分类任务上使用DistilBERT架构，硬掩码和软掩码策略分别达到0.8200和0.8165的准确率，与全注意力的0.8200持平，但显著减少了理论计算开销。该方法为构建更高效、可解释且融入语言学知识的Transformer模型提供了新路径。
Grammatically-Guided Sparse Attention for Efficient and Interpretable Transformers
The quadratic complexity of self-attention in Transformer models remains a significant bottleneck for processing long sequences and deploying large language models efficiently. For this approach, there has been significant research into Sparse Attention, and Deepseek Sparse Attention has combined various methods of creating segments of tokens to reduce the time complexity. This paper introduces a novel approach, Grammatically-Guided Sparse Attention, which constrains attention computations based on the grammatical roles of tokens. By leveraging Parts-of-Speech (POS) tags, attention masks are dynamically generated that enforce linguistically coherent connections between tokens, reducing the computational graph without sacrificing essential linguistic dependencies. Two masking strategies are proposed and evaluated: a hard mask that strictly allows only predefined grammatical interactions, and a soft mask that biases attention towards these interactions. The experiments, conducted on the SST-2 sentiment classification task using a DistilBERT-like architecture, demonstrate that Grammatically-Guided Sparse Attention maintains comparable accuracy to full attention while significantly reducing the theoretical computational overhead. Preliminary results show accuracy values of 0.8200 for hard masking and 0.8165 for soft masking, closely matching the 0.8200 of full attention, providing a path towards more efficient, interpretable, and linguistically-informed Transformer architectures.