FuTransformer: FP8 prefill and INT4 decode kernels for Hopper GPUs
FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
The paper shows the quality-price of aggressive quantization is now measured, not guessed: INT4 weights plus a Hadamard-rotated INT4 cache cost under one perplexity point while roughly tripling decode speed on H100. Engineers serving Llama-3.1-class models on Hopper GPUs can use its fragment-layout dequantization and KV-cache placement techniques to cut latency without falling back to padded prefill. Compared with PyTorch's INT4 kernel, which loses to BF16 at batch 32, this design stays fast up to batch 32.
A new arXiv paper presents FUTRANSFORMER, a CUDA implementation of a Llama-style Transformer block for NVIDIA Hopper GPUs, combining FP8 prefill with INT4 decode. On one H100, prefill reaches 1.29 PFLOP/s and runs up to 1.48x faster than compiled PyTorch FP8, while the FP8 path is 2.02-2.18x faster than the evaluated BF16 CODA path. INT4 decode is 1.3-2.4x faster than PyTorch's best BF16 or INT4 block across 8B and 70B shapes at batch sizes 1-32, and complete Llama-3.1-8B decode takes 2.8 ms per step at batch one versus 8.1 ms for the compiled transformers implementation. Accuracy cost is quantified: decoding WikiText-2 through the INT4 path raises perplexity from 6.24 to 7.23 with round-to-nearest weights, 6.99 with GPTQ, plus 0.20 from the rotated INT4 cache.
FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
Transformer prefill reuses weights across tokens and is dominated by matrix multiplication and activation data movement; single-token decode has limited reuse and is dominated by weight and KV-cache reads. We present FUTRANSFORMER, a CUDA implementation of a Llama-style Transformer block for NVIDIA Hopper GPUs. It applies CODA's GEMM-plus-epilogue approach to inference, adding direct KV-cache placement, folding the fixed RMSNorm gain into weights, writing FP8 output directly with delayed scaling, and supporting prefill without padding. Decode uses INT4 projection weights and an INT4 KV-cache copy. Weights and keys follow Tensor Core fragment layouts for direct dequantization into MMA operands without full intermediate tiles. A block-diagonal Hadamard transform reduces cache quantization error from outlier channels. On one H100, prefill is up to 1.48x faster than compiled PyTorch FP8 and reaches 1.29 PFLOP/s. For gate/up projection with SwiGLU at the 8B shape, our BF16 path is 1.13x faster than CODA. Our FP8 path is 2.02-2.18x faster than the evaluated BF16 CODA path. Across 8B and 70B shapes at batches 1-32, INT4 decode is 1.3-2.4x faster than the faster of PyTorch's BF16 and INT4 blocks at each point: 2.1-3.3x faster than compiled, CUDA-graphed BF16 and 1.3-6.3x faster than the same block using PyTorch's INT4 projection kernel, which targets a single query row and loses to BF16 at batch 32. It is also 31-51% faster than our FP8 path. Complete Llama-3.1-8B decode takes 2.8 ms per step at batch one, versus 8.1 ms for the compiled transformers implementation. Decoding every WikiText-2 token through the INT4 weight path raises perplexity from 6.24 to 7.23 with round-to-nearest weights or 6.99 with GPTQ in the same format; the rotated INT4 cache adds 0.20. The kernels match a quantized reference computation within 0.001 perplexity.
