这篇论文展示了PagedWeight方法，能在MoE模型服务里动态量化权重，省内存还保准，比之前的方法效果明显更好。
MoE大模型在KV缓存密集场景下，模型权重与KV缓存之间存在GPU内存竞争。PagedWeight方法在运行时动态量化MoE权重，平衡专家权重精度与缓存大小。在多个内存敏感场景中，PagedWeight在FP16等效精度下节省72.0% GPU内存并提升1.94倍吞吐量。在相似内存预算下，其质量比现有量化方法高出39.3%，吞吐损失不超过4.1%。
PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
Mixture-of-Experts (MoE) is a popular class of large language models (LLMs), offering high efficiency and accuracy. However, in KV-cache-intensive serving scenarios, MoEs often exhibit a tension between the GPU memory requirements of the model weights and the growing KV cache. We propose PagedWeight, a novel management method for MoE LLM serving that dynamically quantizes MoE model's weights at runtime and balances expert-weight precision with the KV cache sizes. PagedWeight exposes and effectively navigates the complex tradeoff between the model's task accuracy, memory consumption, and throughput/latency. Across several memory-sensitive MoE serving scenarios, PagedWeight improves the quality-memory tradeoff over several existing quantization baselines. PagedWeight achieves FP16-equivalent accuracy with up to 72.0% GPU memory savings and 1.94$\times$ throughput improvement, and improves quality over quantization methods by up to 39.3% at a similar memory budget with at most 4.1% throughput loss.