小模型也能提升推理？MADA-RL用LoRA微调1/16参数，让1.5B模型准确率涨2个点，评论家专挑生成器错误来纠正。
MADA-RL是一个针对参数≤4B紧凑模型的后训练框架，通过LoRA微调少量参数，将模型分为生成器和评论家角色。其核心是反事实评论家优势：动态基线使评论家优化方向为改进生成器共识而非简单复现正确答案。在五个数学推理基准上，DeepSeek-R1-Distill-Qwen-1.5B模型准确率从39.9%提升至41.9%（+2.0点，p<0.001），可训练参数仅为全微调基线的1/16。该方法接近但未超越使用更大数据集的DeepScaleR和STILL-3，分析表明训练后的评论家更擅长纠正生成器错误。
MADA-RL: Multi-Agent Debate-Aware Reinforcement Learning for Parameter-Efficient Reasoning in Compact Models
Large language models achieve strong reasoning performance, but often at prohibitive training cost - a challenge that is especially acute for compact models ($\leq 4 \, \mathrm{B}$ parameters) trained under limited budgets. We introduce MADA-RL, a post-training framework that specializes compact models into generator and critic roles and trains them with a debate-aware learning signal, fine-tuning only a small subset of parameters via LoRA adapters. Our central contribution is a counterfactual critic advantage: a dynamic, role-conditioned baseline that redefines the critic's advantage as its reward minus the generator ensemble's per-instance accuracy. This explicitly optimizes critics to improve over generator consensus rather than to merely reproduce a correct answer, yielding more targeted credit assignment than static mean-reward normalization. At deployment, the specialized agents are composed in a lightweight multi-round protocol. Across five mathematical reasoning benchmarks, MADA-RL raises the accuracy of the DeepSeek-R1-Distill-Qwen-1.5B model from $39.9 \, \%$ to $41.9 \, \%$ ($+2.0$ points, $p < 0.001$) using $16$ times fewer trainable parameters than fully fine-tuned baselines, placing it on the accuracy-trainable-parameter Pareto front. It approaches, but does not surpass, the strongest baselines (DeepScaleR, STILL-3), which are trained on substantially larger datasets; we analyse this gap and the associated inference-time cost directly. A controlled study isolates the source of MADA-RL's gains: the counterfactual advantage produces the highest critic improvement rate of any model evaluated, indicating that trained critics learn to correct generator errors rather than to imitate them.