Speculators v0.9.0 Adds Faster Training and FP8 Backend
Faster training and halved storage lower the cost of building custom speculative decoding models, which speed up inference without changing model weights. Teams running vLLM can now train draft models on their own workloads and deploy them in the same stack. The FP8 backend matters most for those serving large models on memory-constrained GPUs.
The Speculators project released v0.9.0, developed with Red Hat AI. The update delivers up to 2x faster forward training passes through a fused loss kernel. It also introduces an FP8 hidden-state backend that cuts storage requirements by 50% without quality loss. Trained speculators can be served directly in vLLM for speculative decoding.
Speculators v0.9.0 is out — faster, lighter speculative decoding training 📷
Up to 2x faster forward training passes with a fused loss kernel, plus an FP8 hidden-state backend that cuts storage 50% with no quality loss.
Train better speculators, serve them in vLLM. Great work @RedHat_AI 📷
