BitDecoding-FP4 Cuts KV Cache to FP4 With 2.4-3.0x Throughput Gains
The co-design removes the trade-off that previously forced a choice between quantization accuracy and decode speed: system-level changes like transposed softmax eliminate lookup overheads that plagued earlier FP4 KV methods. Teams running long-context inference on NVIDIA Blackwell hardware can apply it directly, since the implementation is open-sourced and works across dense and MoE models with GQA and MLA, and it even holds up in RL training rollouts.