想压缩模型又不想掉太多精度？SLORR在训练时加低秩正则化，开销很小，效果很好，值得一试。
SLORR是一种无需修改架构、无需SVD的低秩正则化框架，基于Hoyer稀疏度量和核范数两种变体。在ImageNet-1K上对ResNet-50、ViT-B/16和ViT-L/16进行短时继续训练，对ResNet-18进行预训练，SLORR在引入不到8%训练开销的同时提升了可压缩性。在135M和560M参数的LLM预训练中，SLORR-Hoyer使压缩模型性能显著优于未正则化模型，平均训练开销低于1%。
SLORR: Simple and Efficient In-Training Low-Rank Regularization
Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they often require SVDs of large weight matrices, modify the model architecture (introducing additional trainable parameters), or rely on stateful cached quantities. To address these limitations, we introduce SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes of the regularizers, for which we provide approximation guarantees. We first evaluate SLORR on ImageNet-1K across short-horizon continued training of ResNet-50, ViT-B/16, and ViT-L/16, and pretraining of ResNet-18, where SLORR induces compressibility while introducing less than 8% training overhead. We further evaluate SLORR-Hoyer in LLM pretraining at 135M and 560M scales: SLORR-trained compressed models preserve performance substantially better than unregularized models while adding less than 1% average training overhead.