Hugging Face 展示如何用 32GB 内存跑 180B 参数 MoE 推理
180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
这篇文章教你在 32GB 内存的普通机器上跑 180B 的 MoE 模型,靠的是按需流式加载专家权重。对想在自己笔记本或小服务器上本地玩开源大模型的开发者来说很实用,文中还有 transformers 的可复现代码。
Hugging Face 发布博客,介绍 MoE expert streaming 技术:在仅 32GB 内存(CPU RAM)的机器上运行 180B 参数的 MoE 模型推理。
核心原理是利用 MoE 架构稀疏激活的特性——每个 token 只触发少量专家权重,因此不必把全部 180B 参数载入内存,而是按需从 SSD 流式加载当前 token 命中的专家分片到 GPU。文中给出基于 transformers 库的完整配置和代码示例,并用大 batch 的持续换入换出(streaming)来掩盖磁盘读取延迟。
局限在于吞吐对磁盘速度和 batch 形态敏感,适合本地低成本部署与实验场景,不适合追求高并发低延迟的生产服务。
This report is categorized under AITOP's "How-to" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include MoE, transformers, Hugging Face, open-source models.
Primary source verification: published directly by Hugging Face: Blog with first-party authority.