斯坦福团队揭示稀疏专家模型路由层间共享几何结构，能预测跨层路由行为，提升模型效率。
研究人员发现稀疏专家模型各层路由状态共享共同几何结构，通过广义正交Procrustes分析对齐后，单个线性转换可达到0.39-0.71的R²值，保留79-90%的预测能力。残差表示跨层更易预测，而路由控制状态更忠实保留模型专家选择。使用预测的规范状态替代原生路由状态，在OLMoE上降低ΔNLL 15.7%，在Phi上10个路由器范围内降低6.2%。
Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts
Sparse mixture-of-experts (MoE) models use an independently parameterized router at each sparse layer to select experts for every token. Prior work has shown that routing decisions across depth can often be predicted from earlier routing signals, suggesting that routing is not fully independent across layers. However, the structure behind this predictability remains unclear. In this work, we provide evidence that routing-relevant states across layers share a common geometric structure that is obscured by layer-specific coordinate systems. We isolate the control subspace of each router and align these spaces into a shared canonical representation using generalized orthogonal Procrustes analysis. After alignment, a single linear transition reaches $R^2=0.39$--$0.71$ and retains 79--90\% of the predictive power of separately fitted layer-specific dynamics, indicating that much of routing-state evolution follows a reusable process across depth. We then ask whether this shared dynamics is specific to routing or simply reflects the smooth evolution of hidden representations. A matched-rank comparison shows that residual representations are often easier to predict across layers, while router-control states preserve the model's expert choices much more faithfully. This separates generic cross-layer predictability from routing-specific information. Finally, we test whether the predicted canonical states remain meaningful when used in place of native routing states. The transported states preserve local routing behavior, while learned state evolution reduces $Δ\mathrm{NLL}$ relative to simple persistence by 15.7\% on OLMoE and 6.2\% over a 10-router horizon on Phi.