Dissecting GPU Utilization for LLM Inference on Nvidia Hopper

It finds that a single utilization percentage can be misleading, especially during decode when small-batch processing leads to underutilization of matrix fragments.

The study profiles vLLM with FlashAttention-3 and cuBLASLt on an H100, replacing a single utilization metric with eight counter-validated views to map utilization gaps to specific mechanisms like fragment fill and kernel selection.

A single SM utilization percentage can make an LLM inference workload look compute-saturated while hiding how much useful work is being done. The problem is not that the counter is wrong, but that it collapses several different mechanisms into one number. This is most severe during decode, where each request contributes only one new token and dense projection GEMMs become small-row matrix multiplications. On Hopper, the bfloat16 GMMA path executes these operations in fixed 64-row matrix fragments, so small-batch decode can fill only a small fraction of each fragment with real token rows. In this paper, we profile vLLM with FlashAttention-3 and cuBLASLt on an H100 NVL across cold prefill, warm prefill, and decode, sweeping sequence length and batch size. We replace the usual single utilization number with eight counter-validated views derived from raw Nsight Compute reports, each pinned to an NCU counter or explicit formula. Together, these views map utilization gaps to concrete mechanisms - fragment fill, occupancy limits, stall signatures, wave quantization, and kernel selection - across four production models and six per-layer kernel roles.

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LLM inference, Nvidia Hopper, GPU utilization, vLLM.

Source attribution: curated from arXiv cs.LG (@Mohammad Siavashi, Gerald Q. Maguire, Dejan Kostic, Marco Chiesa). Cross-source corroboration: 2 independent sources have covered this development (including 宝玉), confirming industry-wide signal strength.