这篇论文把SMoE的不连续性分析得清清楚楚，还附带了一个计算量很小的平滑方法，实验证明又能提高效果，值得一看。
稀疏混合专家（SMoE）架构通过Top-k专家选择实现条件路由，但这会导致映射不连续。论文将不连续性按阶数分类（由切换事件中并列专家数决定），并建立渐近体积估计：低阶不连续性占据主导，高阶体积趋于零。通过扩散过程建模输入随机扰动，证明路径几乎必然在有限时间内首次击中一阶不连续性。实验结果在语言和视觉任务上表明，提出的平滑机制不仅恢复连续性，还提升了经验性能。
Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts
Sparse Mixture-of-Experts (SMoE) architectures are now widely deployed in state-of-the-art language and vision models, where conditional routing allows scaling to very large networks. However, this very Top-$k$ expert selection that enables conditional routing also renders the SMoE map inherently discontinuous. In the vicinity of these discontinuity surfaces, even inputs that are arbitrarily close may activate substantially different sets of experts resulting in significantly different outputs. In this work we give a rigorous geometric and stochastic analysis of these discontinuities. We first classify them by order, determined by the number of tied experts at a switching event. Using measure-theoretic slicing arguments, we establish asymptotic volume estimates for the thickened discontinuity surfaces, showing that lower-order discontinuity sets dominate, whereas higher-order ones occupy a vanishingly small relative volume. Next, modeling random perturbations in the input space via a diffusion process, we prove that the path eventually encounter a discontinuity, and moreover that the first hit almost surely occurs on an order-1 discontinuity with explicit finite-time probability bounds. We further derive occupation-time bounds that quantify the duration the random path spend in the neighborhoods of each discontinuity order. These theoretical results imply that inputs are more likely to lie near lower order discontinuities. Motivated by this insight, we propose a simple smoothing mechanism that can be directly applied to existing SMoEs, softly incorporating experts near discontinuities; our analysis guarantees that the added computational overhead remains small while providing localized smoothing near discontinuities, and experiments across language and vision tasks show that smoothing not only enforces continuity of the SMoE map but also enhances empirical performance.