JD ROAD MOPD refines multi-teacher distillation to cut gradient conflict
The frozen reference model makes agreement checks brittle: if it scores a token lower than both teachers, the teachers count as "agreeing" even when their gap is large, and in-between scores flip the logic. That limitation matters for anyone building multi-teacher distillation pipelines and evaluating whether relative-probability gating or explicit KL divergence between teachers gives a cleaner compatibility signal. Chinese lab output on distillation tricks like this tends to preview what lands in later open-source model training recipes.
JD proposes Cross-Teacher Guarding for MOPD, an on-policy distillation refinement that assigns each student response both a primary teacher and an auxiliary teacher. The two teachers' predictions are compared relative to a frozen reference model to judge whether they agree. When the teachers agree on a token, supervision is strengthened; when they disagree, it is reduced, suppressing gradient conflict from teacher divergence. The method is presented in a 2 Sep 2026 social media post by researcher teortaxesTex, who argues a KL-based agreement signal might outperform the frozen reference model.
from JD, a refinement on MOPD to further suppress gradient conflict due to [mostly superficial] teacher divergence: Cross-Teacher Guarding for MOPD (ROAD MOPD) -assigned teacher and an auxiliary teacher evaluate a student response. - Their predictions are compared relative to a frozen reference model - When teachers agree, strengthen the token's supervision; when disagree, reduce
I see the utility, but it's obviously hobbled by the reference model. Eg if it assigns lower prob than both teachers, then they "agree", and if it's in between, then not. The conceit here is that agreement *relative to the student's initialization* is a viable proxy for cross-domain compatibility. OK but I suspect that some KL-based approach would be better
