这篇论文把MoE的校准问题研究透了，告诉你硬路由和软路由有什么区别，还给了一个对抗性重加权的解法。
该论文研究混合专家模型在分布偏移下的校准问题。先前工作表明专家级校准可提升MoE模型的准确率和校准度。本文发现硬路由模型中专家校准足以保证整体校准，但软路由模型则不足。对此提出对抗性重新加权方法，惩罚分布偏移下路由聚合的校准误差。实验证明该方法能改善平均及困难子集上的准确率-校准权衡。
Toward Calibrated Mixture-of-Experts Under Distribution Shift
Calibration aligns a model's predictive uncertainty with the frequencies of its empirical outcomes and is important for understanding and trusting reported probabilities. Recent work shows that enforcing calibration at the level of individual predictors can improve ensemble accuracy and calibration, with mixture-of-experts (MoE) models showing strong empirical improvements in particular; however, the conditions under which calibration helps MoE are not well understood. In this work, we study how MoE models behave under distribution shift, focusing on how routing mechanisms interact with expert-level calibration. We show that expert calibration is sufficient to ensure calibration of the overall model under a broad class of distribution shifts in hard-routed models, but is insufficient for calibrating soft-routed models. To address this, we propose an adversarial reweighting that penalizes calibration errors of the routed aggregate under distribution shift, and we demonstrate that it improves the accuracy-calibration tradeoff both on average and on difficult subsets of the data, across model classes, prediction tasks, and distribution shifts.