新方法让31.5B参数模型训练省1/3时间，性能还更好，适合大规模MoE模型训练。
研究人员提出通信高效专家混合模型(CE-MoE)，采用异构层模式解耦令牌混合和通道混合深度。在2B至31.5B参数规模测试中，CE-MoE模型在匹配总参数和激活参数的情况下，减少了33.3%的GPU训练时间，同时保持验证损失和下游基准性能与全MoE基线相当。31.5B规模时，CE-MoE还提升了平均下游分数和推理吞吐量。
Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration
When training Mixture-of-Experts (MoE) language models with expert parallelism, all-to-all token dispatch and combine collectives can consume a substantial fraction of end-to-end training time. In this work, we study communication-efficient MoE models (CE-MoE), in which we adopt a heterogeneous layer pattern that decouples token-mixing and channel-mixing depth. Compared to conventional models which interleave MoE layers after each token-mixing layer (e.g., attention, Mamba-2), CE-MoE models concentrate expert capacity in a select few routed MoE layers, while maintaining depth by adding additional token-mixing and dense-FFN layers. Across a scaling ladder from 2B to 31.5B total parameters, under matched total and activated parameters, CE-MoE models consistently reduce training cost while matching validation loss and downstream benchmarks with full-MoE baselines. At the 31.5B scale, CE-MoE uses 33.3\% fewer GPU-hours while improving average downstream score and inference throughput.