这篇论文告诉你：把DeepSeek-R1的推理能力蒸馏到小模型Qwen2.5-7B上，MATH-500能到73%，但回答越短准确率掉得越快。
该论文研究从大型推理模型DeepSeek-R1蒸馏知识到紧凑学生模型Qwen2.5-7B。使用2011-2025年约翰·奥布莱恩数学竞赛问题构建CoT训练语料，通过LoRA在Apple Silicon上微调。基础Qwen2.5-7B准确率64.67%，教师DeepSeek-R1达91.40%。五次独立微调后学生模型平均准确率69.43%（标准差0.17%），比基础模型提升4.76个百分点，并在MATH-500上泛化至73.1%（标准差0.18%）。研究还发现回答长度从平均220词降至31.2词时，准确率从69.43%降至41.9%。
Knowledge Distillation from Large Reasoning Models to Compact Student Models: A Case Study on the John O Bryan Mathematics Competition
This paper investigates knowledge distillation from a large reasoning model (DeepSeek-R1) to a compact student model (Qwen2.5-7B). Using historical problems from the John O'Bryan Mathematics Competition at Northern Kentucky University (2011-2025), we build a Chain-of-Thought (CoT) training corpus through a dual-agent framework. The dataset is used to fine-tune the student model with Low-Rank Adaptation (LoRA) on Apple Silicon hardware using the MLX framework. The base Qwen2.5-7B model achieves 64.67% accuracy on competition problems, while the DeepSeek-R1 teacher achieves 91.40%. An initial 1,000-iteration training run revealed severe overfitting, with validation loss reaching a minimum at iteration 200 before rising steadily. Based on this finding, we ran five independent training runs each limited to 200 iterations with varied random seeds to assess result stability. Across these five runs, the fine-tuned student model achieves a mean accuracy of 69.43% (std dev 0.17%) on the competition dataset, a 4.76 percentage-point improvement over the base model, and generalizes to 73.1% (std dev 0.18%) on the MATH-500 benchmark. We further study how response length affects answer quality across six reasoning levels (R1-R6): accuracy declines consistently from 69.43% at R1 (mean 220 words) to 41.9% at R6 (mean 31.2 words), with the two-person speed section most sensitive to token reduction. These results demonstrate that CoT distillation improves compact student models and that response length is a critical factor in mathematical reasoning quality.