这篇论文把MoE模型卸到边缘设备上跑，靠专家相似性分组减少跨服务器通信，实测延迟和流量都降了，质量只掉一点。
OrderMoE提出了一种利用专家相似性的分布式部署框架，专为资源受限的边缘基础设施设计。它通过路由器诱导的logits表示构建专家相似性模型，将MoE层中的专家分组到多个相似性组中。然后采用相似性感知的专家分组与部署策略，提高边缘服务器间的本地相似性覆盖率。在真实边缘测试平台上，OrderMoE显著降低了平均延迟、尾延迟、跨服务器流量和远程专家调用比例，且推理质量下降可控。
OrderMoE: An expert similarity driven distributed edge MoE inference
Although mixture-of-experts, MoE, models have been increasingly adopted to scale large language models with moderate computation cost, it remains challenging to deploy MoE inference over resource-constrained and bandwidth-limited edge infrastructures. Existing distributed MoE serving methods mainly rely on exact expert placement, caching, replication, or communication scheduling, while overlooking the functional similarity among experts, which provides an opportunity to reduce cross-server token transmission. Therefore, this paper introduces a similarity-aware expert allocation and distributed deployment framework, dubbed OrderMoE, which aims to accelerate edge MoE inference while balancing inference latency, communication overhead, server workload, and inference quality. OrderMoE first constructs an expert similarity model based on router-induced logits representations and partitions experts in each MoE layer into multiple similarity groups. Then, it develops a similarity-aware expert grouping and deployment strategy to improve local similarity coverage across edge servers. Since reducing remote expert invocation and preserving exact inference quality are conflicting objectives, OrderMoE further designs a quality-aware and trajectory-aware runtime server-expert selection algorithm to decide whether a token should invoke its remote target expert or use a feasible local substitute expert. Experimental results on a real distributed edge testbed show that OrderMoE significantly reduces average latency, tail latency, cross-server traffic, and remote expert invocation ratio, while introducing only small and controllable inference quality degradation.