FP4 FlashAttention-4 Hardware Optimization
Hardware-Aware FP4 FlashAttention-4
NVIDIA GB200 users gain 2.13× faster attention inference and 1.14× faster training updates with this optimization.
Blackwell's FP4 tensor cores face challenges with softmax conversion and on-chip dependencies.
Direct-P achieves up to 2.13× BF16 forward throughput on NVIDIA GB200 for noncausal inference.
The causal path uses FP8 gradient operands, accelerating 8B parameter updates by 1.14× on a single GPU.
Hardware-Aware FP4 FlashAttention-4
Blackwell's 4-bit floating-point (FP4) tensor cores do not automatically make attention faster because softmax conversion and on-chip dependencies dominate once its matrix products shrink. We address this with \emph{Direct-P} for noncausal inference and a causal path that passes the forward quantization directly into backward. Direct-P maps scores directly to FP4 probabilities and reaches up to 2.13$\times$ the bfloat16 (BF16) forward throughput on an NVIDIA GB200. The causal path reconstructs probabilities from saved quantized queries and keys and uses 8-bit floating-point (FP8) gradient operands, accelerating a complete single-GPU 8-billion-parameter update by up to 1.14$\times$. Matched distributed training retains FP8 probabilities and values; every tested MXFP4 probability/value training trajectory diverges.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include FlashAttention-4, FP4, NVIDIA GB200, tensor cores.
Source attribution: curated from arXiv cs.LG (@Robert Hu).