这篇论文用剪枝把BERT砍到只剩10%参数，孟加拉语任务上还能和大模型打平，做低资源语言NLP的可以看看。
该论文提出BnBERT-iPET，一种面向孟加拉语的稀疏少样本语言建模方法。通过迭代模式利用训练和彩票假设剪枝，模型仅保留初始BERT网络10%的边，实现90%稀疏度。在孟加拉语标准基准数据集上，该轻量模型与Bangla Electra、Indic-BERT、XLM-RoBERTa等大规模模型表现相当。该方法降低了资源受限语言训练大模型的计算成本和碳排放门槛。
BnBERT-iPET: Sparse Few-Shot Language Modeling for Bengali via Lottery Ticket Pruning
Deep neural networks have shown impressive success in NLP tasks owing to their complex structure and huge number of edges. Achieving state-of-the-art performance in natural language processing with a large pre-trained model such as BERT is expensive and time-consuming, carries a large carbon footprint, and is difficult to realize on machines with minimal computational capability. This creates a barrier to training complex models for resource-constrained languages such as Bengali. However, in a complex neural model, not all edges are equally impactful, and the contributions of some of them can be neglected. Pruning promises to reduce the memory footprint of regular networks, shorten the training time of ever-growing networks, and increase inference efficiency without sacrificing comparable performance. In this work, we introduce BnBERT-iPET, a sparse few-shot language modeling approach for Bengali, and experimentally show that a lightweight few-shot-learned language model retaining only 10% of the edges of an initial model such as BERT can perform neck and neck with much larger models on challenging tasks for a resource-constrained language such as Bengali. By learning from few shots through iterative pattern exploiting training and achieving 90% sparsity with the Lottery Ticket Hypothesis pruning technique, our pruned BnBERT-iPET model proves to be a tough competitor to state-of-the-art language models such as Bangla Electra, Indic-BERT, and XLM-RoBERTa on downstream tasks over standard benchmark datasets of the Bengali language.