WISD：用弱化模型做自蒸馏提升推理能力
Weaken to Strengthen: Weakening-Induced On-Policy Self-Distillation for Large Reasoning Models
这篇讲了个反直觉思路：把模型故意剪枝或量化变弱，再用强弱对比来自蒸馏，Qwen3.5-4B 在 HMMT 上直接涨了 12.8 个点。
论文提出 WISD 方法，通过构造原模型的弱化版本（低比特量化或幅度剪枝），利用强弱对比作为训练信号进行自蒸馏。该方法不需要特权信息，只需无标注提示词。剪枝将 Qwen3.5-4B 在 HMMT 2025 上的准确率从 57.7% 提到 70.5%，在 LiveCodeBench v6 上从 39.1% 提到 56.7%。量化将 Qwen3.6-35B-A3B 在同一基准上分别从 68.5% 提到 74.6%、61.8% 提到 69.5%。量化与剪枝平均比原模型提升 4.0 和 5.5 个百分点。
Weaken to Strengthen: Weakening-Induced On-Policy Self-Distillation for Large Reasoning Models
Self-distillation offers a way to improve reasoning models when a suitable external teacher is unavailable. However, common approaches rely on privileged information to construct a stronger self-teacher, limiting their use when only unlabeled prompts are available. In this paper, we introduce Weakening-Induced On-Policy Self-Distillation (WISD), which instead constructs a weaker counterpart of the original model and uses the contrast between the two as a training signal without requiring privileged information. We explore two ways to construct the weaker model: low-bit quantization and magnitude pruning. We evaluate WISD across diverse architectures and scales on mathematical and coding reasoning tasks. Across these benchmarks, quantization and pruning improve average accuracy over the original models by 4.0 and 5.5 percentage points, respectively, achieving state-of-the-art performance even without privileged information. Notably, pruning raises Qwen3.5-4B's accuracy from 57.7% to 70.5% on HMMT 2025 and from 39.1% to 56.7% on LiveCodeBench v6. Quantization raises Qwen3.6-35B-A3B's accuracy from 68.5% to 74.6% and from 61.8% to 69.5% on the same benchmarks, respectively. These results establish WISD as a novel and effective self-distillation framework for large reasoning models.
- rohanpaul_ai10-27 17:51原文
