HyperParallel-MoE：在昇腾NPU上实现快速MoE训练的多核交错调度
HyperParallel-MoE: Multi-Core Interleaved Scheduling for Fast MoE Training on Ascend NPUs
对于在昇腾NPU上训练大规模MoE模型的团队，HyperParallel-MoE通过细粒度异构调度解决了通信与计算重叠不足的痛点，实测延迟降低1.58倍，值得关注并尝试集成到现有框架中。
本文提出了HyperParallel-MoE，一个针对昇腾NPU的MoE训练编译与调度框架。现有框架在昇腾NPU上串行执行MoE算子，未能充分利用其异构计算资源（矩阵AIC和向量AIV）。
HyperParallel-MoE将算子级执行转化为静态调度的tile级异构任务流，通过AIV驱动单边通信消除主机侧同步，并实现通信与计算在单个内核启动内的细粒度重叠。在MindSpore和MindFormers栈中实现，基于DeepSeek风格MoE模型在昇腾A3集群上测试，将Dispatch-to-Combine的MoE-FFN延迟最多降低1.
58倍。这项工作表明tile级异构调度能显著提升现代NPU上的MoE训练效率。
HyperParallel-MoE: Multi-Core Interleaved Scheduling for Fast MoE Training on Ascend NPUs
Modern Mixture-of-Experts (MoE) models increasingly rely on large-scale AI accelerator clusters for efficient training. Ascend NPUs expose heterogeneous on-chip compute resources, including matrix-oriented AIC units and vector-oriented AIV units with explicit cross-queue synchronization support. However, existing training frameworks largely execute MoE operators in a serialized kernel-by-kernel manner, leaving substantial heterogeneous parallelism underutilized. This paper presents HyperParallel-MoE, a compilation and scheduling framework for MoE training on Ascend NPUs. HyperParallel-MoE transforms operator-level MoE execution into a statically scheduled tile-level heterogeneous taskflow spanning AIC and AIV resources. It introduces AIV-driven one-sided communication to eliminate host-side collective synchronization, dependency-preserving tile task generation to unify communication and computation under a common task abstraction, and event-driven static scheduling to coordinate cross-queue execution with low runtime overhead. HyperParallel-MoE further executes the compiled taskflow within a unified runtime that concurrently drives AIC and AIV workers inside a single kernel launch, enabling fine-grained overlap among communication, matrix computation, and vector computation while preserving existing optimized operators. We implement HyperParallel-MoE in the MindSpore and MindFormers stack and evaluate it using DeepSeek-style MoE models on Ascend A3 clusters. Across multiple expert-parallel configurations, HyperParallel-MoE reduces Dispatch-to-Combine MoE-FFN latency by up to 1.58x, demonstrating that tile-level heterogeneous scheduling can substantially improve MoE training efficiency on modern NPUs.
This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Key entities and technical topics under continuous tracking include MoE训练, 昇腾NPU, 异构调度, 编译优化.
Primary source verification: published directly by arXiv: DeepSeek with first-party authority.
