CECAR 用缓存感知路由让 MoE 大模型在 16GB 显存显卡上跑出 12.5 tokens/s
CECAR: Cache & Expert Co-Aware Routing Accelerates On-Device Inference of MoE LLMs
这篇论文有意思的地方是换了思路:不压缩模型,而是预测专家的重用距离来决定路由。Qwen3-30B-A3B 在 16GB 显存的显卡上跑出 12.5 tokens/s,比把模型塞满内存还快。做端侧部署、推理框架的工程师可以直接看开源代码。
arXiv 论文提出 CECAR(Cache & Expert Co-Aware Routing),面向 MoE 大模型的端侧推理。在 Qwen3-30B-A3B 上,CECAR 在仅有 16GB 显存的消费级 GPU 上达到 12.5 tokens/s,非驻留专家权重从 SSD 按需读取,相比传统 MoE 推理加速 4.5 倍,解码速度还超过了把整个模型完整放进 DRAM 的基线方案。
技术上,CECAR 联合考虑专家路由和缓存管理。作者先给出三个观察:即使采用 Belady 最优替换策略,缓存命中率也有上限;该上限不足以支撑低延迟端侧推理;此前基于二值缓存命中与否做路由的方法会导致路由不稳定。CECAR 用机器学习策略预测每个专家的期望重用距离(reuse distance),把路由和执行优先导向重用距离更小的专家,从源头减少昂贵的 SSD 换页。
对想在本地或边缘设备部署 Qwen3-30B 这类 MoE 模型的开发者,这项工作提供了一条不做量化压缩也能跑大模型的路径,代码已在 GitHub 开源。相比单纯扩内存或蒸馏小模型,它把瓶颈从硬件容量转移到调度策略上,对端侧 AI 硬件厂商和推理框架作者都有参考价值。
CECAR: Cache & Expert Co-Aware Routing Accelerates On-Device Inference of MoE LLMs
Despite growing interest in on-device LLM deployment, large-scale Mixture-of-Experts (MoE) models remain impractical on resource-constrained devices, as sparse expert activation still requires all expert weights to be memory-resident. To address this, we identify three key observations for on-device MoE inference: (a) cache hit rates are fundamentally bounded even under Belady's optimal policy; (b) these bounds are insufficient for low-latency on-device inference; and (c) prior cache-aware routing based on binary cache presence can induce routing instability. Based on these insights, we propose CECAR (Cache & Expert Co-Aware Routing), a MoE inference system that jointly considers expert routing and cache management. Unlike prior cache-aware routing methods, CECAR uses an ML-based cache policy to predict the expected reuse distance of each expert and prioritizes routing and execution toward experts with smaller predicted reuse distances. On the Qwen3-30B-A3B model, CECAR achieves 12.5 tokens/s on a consumer-grade GPU with only 16 GB of VRAM, where non-resident experts are fetched from SSD, delivering a 4.5x speedup over conventional MoE inference and exceeding the decode speed of baselines that keep the entire model fully resident in DRAM. Code is available at https://github.com/hoonki2002/CECAR.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Qwen3, MoE, inference optimization, open-source models.
Source attribution: curated from arXiv cs.AI (@Byeongju Kim, Hoonki Lee, Byungjun Kim, Inyul Ra, Yongjin Kim, Yongseong Lee, Sangyeob Kim).