MoE-CORE System for Memory-Constrained Inference
MoE-CORE: Coordinated Expert Offloading and Residency for Memory-Constrained MoE Inference
The system dramatically reduces inference latency by 96% compared to existing methods, making MoE models practical on memory-constrained devices. Developers working with large language models can implement this approach to deploy MoE architectures on compact hardware without significant performance degradation, potentially enabling new applications in edge computing environments.
MoE-CORE coordinates expert offloading and residency for memory-constrained MoE inference.
It uses alternating buffers during prefill and combines nonuniform layer-wise cache capacity, domain-informed initialization, and cross-layer prefetching during decode.
On DeepSeek-V4-Flash-W4A8, MoE-CORE achieves a mean time per output token of 38.0-44.
8 ms versus 1268.9-1269.
1 ms for vLLM Prefetch.On GLM-5.
2-W4A8C8, the values are 206.6-220.
5 ms versus 5941.5-5941.
8 ms.
MoE-CORE: Coordinated Expert Offloading and Residency for Memory-Constrained MoE Inference
Sparse expert activation reduces MoE models' computation, yet expert weights can exceed limited device memory. Offloading makes inference feasible on a compact AI appliance but exposes host-to-device transfers to the inference path. We present MoE-CORE, a system that coordinates expert offloading and residency for memory-constrained MoE inference. It stages complete expert layers in alternating buffers during prefill. During decode, it combines nonuniform layer-wise cache capacity, domain-informed initialization, routing-history-aware replacement, and cross-layer prefetching. The main configuration executes router-selected experts exactly; an optional score-based substitution path handles eligible low-score misses. The main comparison uses 1K- and 128-token output caps for MoE-CORE and vLLM Prefetch, respectively. Across five workloads per model, MoE-CORE records a mean time per output token (TPOT) of 38.0-44.8 ms versus 1268.9-1269.1 ms for the evaluated vLLM Prefetch configuration on DeepSeek-V4-Flash-W4A8; the corresponding values on GLM-5.2-W4A8C8 are 206.6-220.5 and 5941.5-5941.8 ms. Under an 84-GB NPU-memory cap, the best measured DeepSeek GSM8K configuration achieves a TPOT of 21.5 ms with approximate expert substitution and multi-token prediction (MTP) at depth 2. These results support coordinated expert residency and transfer scheduling under a device-memory constraint. The code is here.
This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include MoE-CORE, DeepSeek-V4-Flash-W4A8, GLM-5.2-W4A8C8, reasoning models.
Primary source verification: published directly by arXiv: DeepSeek with first-party authority.