这个工作很实在：跨层共享专家参数让MoE模型内存减半，性能不掉，适合做模型压缩和高效训练的朋友看看。
混合专家架构通过每个token激活少量专家来高效扩展大语言模型，但训练和推理时仍需加载全部专家参数。本研究提出Expert Tying方法，在保持独立层路由和注意力的前提下，将相邻Transformer层的专家参数共享。在OLMoE、Qwen3和DeepSeek-style MoE上的预训练实验显示，该方法可将内存占用减少近2倍，且困惑度和下游任务质量几乎不受影响。该方法利用了MoE路径中固有的参数冗余，实现了计算与内存的高效权衡。
Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models
Mixture-of-Experts (MoE) architectures efficiently scale Large Language Models (LLMs) by activating only a small fraction of their experts per token, yet the full parameter count - dominated by the expert parameters - must be held in training and inference memory. To address this, we introduce Expert Tying, an architectural modification that shares expert parameters across consecutive transformer layers while preserving independent, layer-wise routing and attention. We evaluate this approach across common, state-of-the-art architectures, including OLMoE, Qwen3, and DeepSeek-style MoEs. Our pretraining experiments demonstrate that tying experts can reduce memory footprint by almost 2x at virtually no degradation in perplexity or downstream quality. By exploiting the parameter redundancy inherent in MoE pathways, our method provides a highly favorable compute-to-memory trade-off, advancing efficient training and scaling of next-generation LLMs.