ModelsSelected

DSpark speculative decoding for Kimi K3.1 Pro cuts draft KV memory

Why we picked it

Bounded draft KV memory removes the main scaling blocker for speculative decoding on million-token contexts, where draft caches previously grew with sequence length. Teams serving Kimi K3.1 Pro on vLLM, especially with P/D disaggregated deployments, can swap in the released weights and config for immediate throughput gains. Compared with the preview draft model, the 30% acceptance-length improvement compounds with the fixed 2K window into a materially cheaper long-context serving setup.

Inferact and Red Hat's AI Speculators team have released DSpark, a new speculative decoding draft model for Moonshot (Kimi)'s Kimi K3.1 Pro, built on the vLLM and Speculators stack. On HumanEval, DSpark achieves up to 30% higher acceptance length compared with the earlier preview version, measured with k=7 draft tokens at temperature 1.0 and top_p 0.95, including the bonus token.

Body

The key architectural change is a 2K sliding window in every draft layer, which keeps the draft model's KV memory bounded even when the target model runs at 1M-token context. Instead of caching the full draft KV state as context grows, each draft layer only retains a fixed 2,048-token window, decoupling draft memory from total sequence length. vLLM's redesigned KV management supports this, including prefill/decode (P/D) disaggregation and various parallelism configurations, so sliding-window draft models can run in production inference clusters.

Speculative decoding throughput depends on how many draft tokens the target model accepts per step, so higher acceptance length directly translates into lower latency and cost for serving Kimi K3.1 Pro. The weights and vLLM configuration are published, letting teams running long-context workloads adopt the setup directly.

Original · vLLM

More accepted tokens. Bounded draft KV memory. 🔥

The new DSpark for @Kimi_Moonshot’s Kimi K3.1 Pro, jointly developed by @Inferact and the @RedHat_AI Speculators team with vLLM + Speculators, brings:

• Up to 30% higher acceptance length vs. the preview (HumanEval). • A 2K sliding window in every draft layer, keeping draft KV memory bounded even at 1M-token context.

Efficient draft architectures need an inference stack that can serve them. vLLM’s redesigned KV management supports P/D disaggregation and different parallelism configurations, bringing sliding-window draft models into production.

Comparison: DSpark k=7, temperature 1.0, top_p 0.95. Acceptance length includes the bonus token.

Weights + vLLM config below. 👇

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Models" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Kimi, Kimi K3.1 Pro, vLLM, speculative decoding.

Source attribution: curated from vLLM (@vllm_project).