NVIDIA 出了个 NeMo AutoModel，基于 Hugging Face Transformers v5，几行代码就能给 MoE 模型训练加速 3 倍以上，搞大模型训练的值得看看。
NVIDIA 发布了 NeMo AutoModel，基于 Hugging Face Transformers v5 为混合专家 (MoE) 模型提供原生支持。通过 Expert Parallelism、DeepEP 和 TransformerEngine 内核，仅需几行代码即可应用优化。实测显示 NeMo AutoModel 将主流 MoE 模型训练吞吐量提升 3.4 到 3.7 倍。该工具是 NeMo 框架的一部分，专为大规模模型构建设计。
The rise of MoE models introduced new challenges in training, and @huggingface's Transformers v5 bro...
The rise of MoE models introduced new challenges in training, and @huggingface 's Transformers v5 brought first-class support for solving them. Now, NeMo AutoModel builds on top of v5. Part of the NeMo framework for building models at scale, NeMo AutoModel brings optimizations to a broad set of model families through support for Expert Parallelism, DeepEP, and TransformerEngine kernels with a few lines of code. We found NeMo AutoModel brings a 3.4 to 3.7x higher training throughput for popular MoE models. You can read more here: nvda.ws/4eopYty 💬 3 🔄 9 ❤️ 31 👀 3042 📊 9 ⚡