FuTransformer: FP8 prefill and INT4 decode kernels for Hopper GPUs
The paper shows the quality-price of aggressive quantization is now measured, not guessed: INT4 weights plus a Hadamard-rotated INT4 cache cost under one perplexity point while roughly tripling decode speed on H100. Engineers serving Llama-3.1-class models on Hopper GPUs can use its fragment-layout dequantization and KV-cache placement techniques to cut latency without falling back to padded prefill. Compared with PyTorch's INT4 kernel, which loses to BF16 at batch 32, this design stays fast up to batch 32.