验证器在新任务上会拖后腿
论文发现验证器驱动的自DPO方法在视觉语言模型自改进中存在任务特异性问题。在MathVista、MMMU和BLINK上用开源验证器阶梯测试，同一验证器在MathVista上提升Qwen-3-VL-2B学生模型，但在MMMU上验证器准确率降至8%-23%，导致学生模型性能下降3.4-10.9个百分点。该现象在Qwen-2.5-VL-3B上复现。论文给出基于方差定理的机械论解释，指出目标任务验证器质量而非参数量才是关键。
When Good Verifiers Go Bad: Self-Improving VLMs Can Regress on New Tasks
Verifier-driven self-DPO is a common recipe for self-improving production visual-language models. In this setup, a frozen verifier scores candidate generations, the top- and bottom-scoring candidates form a preference example, and DPO updates the learner. The deployment-time assumption is monotone: a stronger verifier should yield a stronger student. We show that this assumption can fail because verifier quality is highly task-specific. On a four-rung open-source verifier ladder across MathVista, MMMU, and BLINK, the same verifiers that are above-threshold and improve a Qwen-3-VL-2B student on MathVista become sub-threshold on MMMU, where their task-rubric accuracy drops to 8% to 23%. In this regime, every verifier we tested silently regresses the student, producing drops of 3.4 to 10.9 percentage points below the frozen baseline while the DPO training loss continues to decrease. The regression replicates on a second student, Qwen-2.5-VL-3B. Moreover, within the failure regime, damage is confidence-inverted: the more accurate-but-still-wrong verifier causes larger regression than a near-random verifier, suggesting that progress-gated replay amplifies confidently wrong preference pairs. We give a compact mechanistic explanation via a variance theorem for progress-gated replay and its direction-mismatch failure mode. The deployment message is operational rather than purely diagnostic: before running any verifier-driven loop, teams should measure target-task rubric accuracy, rank verifiers by target-task rubric quality rather than parameter count, and treat diminishing returns in above-threshold regimes as a verifier-side compute budget cap.