本地跑MoE大模型终于能追上云端的服务质量了——做本地部署的开发者可以直接参考这套CPU-GPU混合方案，不用再忍受降级模型和低吞吐。
本地部署大型混合专家模型（MoE）在服务质量上远不及云端环境，即使低并发场景也存在四大差距：依赖降级模型、无法满足长预填充的30秒TTFT、解码吞吐量低于20 tokens/s、混合负载下并发能力差。本文提出CPU-GPU混合系统，通过流式加载预填充（SLP）将预填充吞吐提升至1200 tokens/s，支持32K提示词在30秒内完成；分布式SLP（DSLP）结合SmallEP专家并行，在双RTX 5090上达到1800 tokens/s和45K提示词；以及零拷贝共享权重的节点内预填充-解码分离、AVX-512优化的FP8 GEMV内核等技术。该系统在消费级CPU-GPU平台上实现了旗舰MoE模型的云级服务质量，无需数据中心基础设施即可获得高质量、低成本的本地推理。
Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design
Local deployment of large Mixture-of-Experts (MoE) models falls short of the service quality achieved in cloud-scale environments, even under low-concurrency workloads. We identify four key gaps in local MoE inference: reliance on capacity-reduced models (quantized, distilled, rerouted), inability to meet 30-second TTFT for long prefills (more than 12K), sub-baseline decode throughput (under 20 tokens/s), and poor concurrency under mixed prefill-decode and batched decode workloads. We present a CPU-GPU hybrid system that achieves cloud-level SLOs on dual-socket commodity CPUs and consumer GPUs by (1) stream-loading prefill (SLP), boosting prefill throughput to 1,200 tokens/s and enabling 32K prompts within 30 seconds; (2) distributed SLP (DSLP) with SmallEP expert parallelism, reaching 1,800 tokens/s and 45K prompts in 30 seconds on two RTX 5090s; (3) intra-node prefill-decode disaggregation with zero-copy shared weights and a dual-batch attention-MoE overlap scheme, sustaining concurrency with under 15 percent latency increase and 50 percent throughput gains; (4) an AVX-512-optimized FP8 GEMV kernel, enabling native CPU FP8 inference while delivering 4-5x lower CPU latency; and (5) fine-grained CPU parallelism that attains 28 tokens/s on INT4 DeepSeek-V3 and 21.5 tokens/s on intact FP8 V3. Evaluations show our system delivers cloud-level QoS for flagship MoE models on consumer CPU-GPU platforms, reshaping local deployment with intact, original-precision inference and enabling high-quality, cost-effective access without datacenter infrastructure.