Cohere paper finds speculative decoding works well on MoE models
Practitioners running low-batch MoE inference, especially agent workloads with single-request traffic, can now justify benchmarking speculative decoding on their own hardware instead of ruling it out based on older cost models. The expert-reuse finding changes how inference engineers should estimate verification costs for open-weight models like Qwen3-235B on H100-class GPUs. Teams serving MoE models at low batch sizes should rerun their own latency tests before committing to a serving strategy.
A new Cohere paper reports that speculative decoding delivers larger-than-predicted speedups on fine-grained mixture-of-experts (MoE) models, even at batch size 1. The team tested 10 MoE models from 30B to 1T parameters on H100 GPUs and found neighbouring tokens share 20% to 45% of their experts, against the 4% to 12% expected from independent routing. For Qwen3-235B in FP8, the verification pass ran at 89% of a normal decoding step's speed, versus 52% predicted. The result contradicts prior work that assumed verifying multiple tokens loads too many extra experts at low batch sizes.
New @cohere paper finds that speculative decoding works well on fine-grained MoE models even at batch size 1, where prior work predicted small gains.
Finds that in mixture-of-experts models, neighboring tokens share 20% to 45% of their experts, against 4% to 12% expected, which makes speculative decoding cheaper than predicted.
Speculative decoding lets a small drafter propose tokens that the big model checks in 1 pass. For MoE models, prior work said checking several tokens loads many extra experts, limiting gains at low batch sizes.
They tested 10 MoE models from 30B to 1T parameters and profiled them on H100 GPUs. Consecutive tokens reuse many experts, and small experts use GPU bandwidth well, so checking extra tokens costs little.
Adjacent tokens shared 20% to 45% of their experts, against 4% to 12% expected from independent routing. For Qwen3-235B in FP8, the verification pass ran at 89% of a normal decoding step's speed, against 52% predicted.
For low-batch MoE serving, such as agent workloads, benchmark speculative decoding on your hardware instead of ruling it out.
