Speculative Decoding for MoE Models: Deduplication and Draft Length
Speculative Decoding for Mixture-of-Experts Models: Expert Bytes, Kernel Bandwidth and the Best Draft Length
Deduplicating expert reads sounds appealing for MoE inference, but the measured bandwidth numbers show it rarely helps at practical draft lengths, which saves practitioners from engineering it blindly. Inference engineers and MoE serving teams can reuse the fitted-bandwidth method, which takes under an hour per model, to predict whether deduplication wins on their own hardware. The tested stacks — llama.cpp CUDA and Metal plus MLX-LM — cover the common Mac and NVIDIA deployment paths.
A paper analyses speculative decoding on mixture-of-experts models, where consecutive tokens often activate the same experts. The authors derive a step-time model based on bytes requested per kernel and effective bandwidth, and prove under geometric acceptance when speculation beats plain decoding, including a monotone staircase behaviour for the best draft length. They modified llama.cpp (Metal and CUDA) and MLX-LM to deduplicate expert reads, finding the per-slot kernel achieves higher effective bandwidth (r = 0.32 to 0.75), so deduplication pays off only at 5 to 8 tokens per pass or never. Predictions from routing traces and fitted bandwidths locate the crossover within one pass size in 10 of 12 configurations, and give the best draft length on CUDA for DeepSeek-V2-Lite within one at all 22 acceptance values. Core proof steps are checked in Lean.
Speculative Decoding for Mixture-of-Experts Models: Expert Bytes, Kernel Bandwidth and the Best Draft Length
Speculative decoding checks several drafted tokens in one verification pass. On a mixture-of-experts (MoE) model, consecutive tokens often select the same experts. A deduplicating engine reads each selected expert once per pass, so it requests fewer bytes than a per-slot engine, which reads it once per token. We write a step's time as the bytes each kernel requests over its effective bandwidth, plus fixed time. Under geometric acceptance, we derive from this step time when speculation is faster than plain decoding. The best draft length is a monotone staircase in the acceptance rate. We bound how a bandwidth error moves each result, and we check the main steps in Lean. Deduplication is faster exactly where its share $\bar u(n)$ of the per-slot bytes is below its ratio $r(n)$ of the per-slot bandwidth. On llama.cpp Metal, MLX-LM and llama.cpp CUDA, which we modified to deduplicate, the per-slot kernel has the higher effective bandwidth (r = 0.32 to 0.75). Thus deduplication is faster only from 5 to 8 tokens per pass, or never. We fit the bandwidths on one layer's expert operation, run alone, in under an hour per model. Bytes from routing traces over these bandwidths predict the time that deduplication saves. The measured saving lies inside its error bound at every point on CUDA and MLX-LM and at 36 of 48 points on Metal. The crossover is the first pass size at which deduplication is faster. The prediction places it within one pass size in 10 of 12 configurations of engine, model and depth. We computed these predictions after the measurements. On DeepSeek-V2-Lite, operation times measured before its speculative runs give the best draft length on CUDA within one at all 22 acceptance values.