Long-Short Contrastive Pruning 让剪枝后的推理模型更省思考 token
Improving the Chain-of-Thought Compressibility of Pruned Reasoning Models via Long--Short Contrastive Pruning
剪枝和 CoT 压缩一起用会互相打架，这篇论文给 DeepSeek-R1-Distill-Qwen-7B 剪 20% 权重还能省 8–22% 思考 token，掉分不到 0.5，做模型部署的可以看看。
论文提出 CSD（Contrastive Set-Difference）方法，通过对比同一问题长短正确答案的神经元激活，优先剪掉与冗余推理相关的神经元。在 DeepSeek-R1-Distill-Qwen-7B 上剪枝 20% 权重后，相比 RAC 剪枝加同样的压缩流程，四个数学基准上思考 token 减少 8–22%。准确率差距控制在 0.5 分以内。该方法还把偏好对扩展到剪枝后模型总是答对或总是答错的问题，弥补了压缩训练的信号盲区。
Improving the Chain-of-Thought Compressibility of Pruned Reasoning Models via Long--Short Contrastive Pruning
Large reasoning models are costly to serve because they are large and because they generate long chains of thought (CoT). Weight pruning and CoT compression address these two costs separately, but they interfere in that pruning adds redundant reasoning on problems the model still solves and incomplete reasoning on those it no longer solves, and on harder problems makes the model less compressible by subsequent length-preference optimization. We trace this to a blind spot on each side: pruning criteria do not distinguish necessary from redundant reasoning, and preference-based CoT compression receives no training signal on problems a pruned model always or never solves. We propose Contrastive Set-Difference (CSD), which identifies neurons enriched in redundant reasoning by contrasting their activations on long and short correct solutions of the same problem and prunes them preferentially, and we extend preference pairs to the problems a pruned model always or never solves. Pruning 20\% of DeepSeek-R1-Distill-Qwen-7B, the combination uses 8--22\% fewer thinking tokens on four math benchmarks than pruning with reasoning-aware calibration (RAC) followed by the same compression, with accuracy within 0.5 points.
