JD ROAD MOPD refines multi-teacher distillation to cut gradient conflict
The frozen reference model makes agreement checks brittle: if it scores a token lower than both teachers, the teachers count as "agreeing" even when their gap is large, and in-between scores flip the logic. That limitation matters for anyone building multi-teacher distillation pipelines and evaluating whether relative-probability gating or explicit KL divergence between teachers gives a cleaner compatibility signal. Chinese lab output on distillation tricks like this tends to preview what lands in later open-source model training recipes.
