How-to81°11月9日 15:17

Hugging Face 发布《The Ultimate MoE Training Handbook》免费开源

来自：https://t.co/c0J98gds7s

AI Will@FinanceYF5

Post

Why we picked it

Hugging Face 花了 80 万 GPU 小时做实验，把 MoE 训练的坑全写成一本书，还带可跑的代码。想自己训 MoE 的直接收藏。

Hugging Face 的 Thomas Wolf 宣布发布《The Ultimate MoE Training Handbook》，耗用 80 万 GPU 小时跑实验整理而成。手册以问题为导向，覆盖专家数量与规模选择、all-to-all 通信隐藏、路由器选型、辅助损失与 DeepSeek 的 loss-free bias 对比等训练细节。全文约 7 万字，含 55 张交互式图表、可运行代码及 Hugging Face 自身实验数据。Wolf 指出当前 20 个最佳开源模型中有 18 个是 MoE 架构，而 MoE 训练经验仍集中在少数团队手中，这本手册旨在公开这些知识。

Original · AI Will

来自：https://t.co/c0J98gds7s

来自： x.com/Thom_Wolf/stat… Thomas Wolf @Thom_Wolf 18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over. And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams. So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit: - how many experts, how big, shared or not? - why is half my step spent in all-to-all, and how do I hide it? - why is my run 25x slower with no error in the logs? - softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32? - aux loss or DeepSeek's loss-free bias? - upcycle a dense checkpoint or train from scratch? - can I serve with fewer experts per token than I trained with? ⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs. Read it here: huggingface.co/spaces/hugging… Your browser does not support the video tag. 🔗 View on Twitter 🔗 View Quoted Tweet 💬 0 🔄 0 ❤️ 0 👀 326 ⚡

#MoE #Hugging Face #DeepSeek #推理模型 #开源模型

Keep reading

Recent stories with the same tag or section

How-to BuildersSelected15:17

Chinese contentHugging Face 投入 80 万 GPU 小时，发布约 7 万字 MoE 训练免费手册

Hugging Face 把 80 万 GPU 小时的 MoE 训练经验整理成 7 万字免费手册，还带 55 张交互图和可运行代码，做训练的可以收藏慢慢啃。

#MoE #Hugging Face #微调 #开源模型

AI Will@FinanceYF5Source

Papers Builders11:09

Chinese contentISER-MoE 通过 NVLink 交换机复制热点专家，提升 MoE 推理吞吐

MoE 推理总被几个热点专家拖慢，这个方案直接借 NVLink 交换机做复制，DeepSeek-V4-Pro 实测数学任务吞吐涨了 17.4%。

#ISER-MoE #MoE #TensorRT-LLM #DeepSeek

Event

arXiv: DeepSeek@Jintao Peng et al. (8)2 sourcesSource

Products Builders23:20

Chinese content高通在第九届进博会展出端侧 MoE 智能体方案

高通把 MoE 智能体直接跑在手机终端上，不靠云端也能多步执行任务，进博会现场有真机演示。

#高通 #MoE #智能体 #端侧AI

量子位@量子位的朋友们Source

How-to BuildersOfficialSelected18:33

DeepSeek, Qwen, Mixtral, Llama 4, gpt-oss MoE Routing Compared

Routing configuration — active parameters, top-k and shared-expert design — directly affects inference cost and throughput, yet specs are scattered across model cards and configs. This consolidation lets engineers pick models by routing efficiency rather than headline parameter counts. It also highlights how differently DeepSeek, Qwen and gpt-oss trade expert granularity against compute per token.

#DeepSeek #Qwen #Llama 4 #gpt-oss

官方一手marktechpost@Michal SutterSource

How-to BuildersOfficial15:11

Chinese contentMoE 路由策略详解：从 Mixtral Top-2 到 DeepSeek-V4 哈希路由

搞 MoE 的朋友可以看这篇，把 Mixtral 的 top-2 到 DeepSeek-V4 的哈希路由等 7 种做法放一起对比，还标了各自用在哪。

#MoE #Mixtral #DeepSeek-V4 #推理模型

官方一手marktechpost@Asif RazzaqSource

Models US vs ChinaSelected01:09

Baidu releases ERNIE-5.5-Flash-Base, a 280B MoE base model

The license excludes the EU, UK and South Korea from its permitted territory, which could push adoption toward markets including Southeast Asia where no restriction applies. Teams fine-tuning base models for reasoning or agentic workloads can compare its KV-cache compression and multi-router MoE against Qwen or DeepSeek bases of similar size. The 1M-token context support for fine-tuning and inference is a practical differentiator.

#ERNIE-5.5-Flash-Base #Baidu #DeepSeek #open-source models

kimmonismus@kimmonismusSource