Inkling 2 Small Runs on vLLM From Day Zero
Day-zero vLLM support plus single-node GB200 fit means operational cost of serving this model is far lower than typical 594B-scale deployments. Engineering teams running self-hosted inference should benchmark the 630 tok/s TP4 figure against their current MoE serving setups, since the DSpark speculator and NVFP4 experts are the levers behind it. Teams needing 1M-token multimodal context without multi-node clusters have a concrete new option to evaluate.
Thinking Machines (@thinkymachines) announced that Inkling 2 Small, its new mixture-of-experts model, is supported by vLLM from day zero of release. The model has 594B total parameters with 17B active per token, accepts both text and image inputs, and supports context lengths up to 1M tokens. It ships with a native DSpark speculator for faster inference.
Architecturally, the MoE experts are quantized in NVFP4 and MXFP8 formats, which lets the full model fit on a single GB200 or GB300 node. At tensor parallelism of 4 (TP4), the model delivers up to 630 tokens per second per user according to the announcement. Day-zero vLLM support means teams can deploy it on standard open-source inference infrastructure without waiting for custom kernels or community adapters.
The combination of a 594B-scale MoE that fits on one node plus high per-user throughput targets self-hosted deployments, where enterprises can run a large multimodal long-context model without multi-node clusters. Support in vLLM, the widely used open-source serving stack, also lowers the integration barrier for developers evaluating the model against other open and proprietary alternatives.
@thinkymachines Inkling 2 Small runs on vLLM from day 0 🚀
594B total / 17B active MoE, text + image in, up to 1M context, with a native DSpark speculator. Experts are NVFP4/MXFP8, so it fits on one GB200/GB300 node.
Up to 630 tok/s per user at TP4.
https://t.co/ptgGjbRNsk https://t.co/xYGsFjbqs5
This report is categorized under AITOP's "Models" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Inkling 2 Small, Thinking Machines, vLLM, open-source models.
Source attribution: curated from vLLM (@vllm_project).
