Epoch AI launches Inference Performance Index for chip comparison
The metric shifts chip evaluation from peak compute to real serving capacity, which changes how buyers should read vendor spec sheets. Infrastructure planners, cloud capacity teams and analysts sizing inference fleets can use IPI scores to model how many concurrent users or agents a given GPU deployment actually supports. Compared with the H100, the GB300's 10× inferred advantage shows why last-generation hardware may be far more constrained for agent workloads than raw FLOP/s suggests.
Epoch AI has introduced the Inference Performance Index (IPI), a new benchmark for AI chips that measures inference performance beyond raw FLOP/s. The organisation estimates that FLOP/s alone understates NVIDIA's GB300 inference advantage over the H100 by a factor of 10×. According to Epoch AI, standard compute-only comparisons miss most of what matters for inference workloads.
The IPI accounts for memory bandwidth and inter-chip communication alongside compute throughput. These three factors together determine how many simultaneous users or agents a chip can support at a given output speed. Scores are normalised into GB300 equivalents, allowing direct comparison between different chip architectures.
The metric has practical applications for infrastructure planning: operators can use GB300-equivalent scores to estimate the total inference capacity of a data centre, or even aggregate global capacity. As inference increasingly dominates AI compute demand with the rise of agentic workloads, compute-only specs have become a misleading basis for procurement and capacity forecasting decisions.
We estimate that FLOP/s alone understates GB300’s inference advantage over H100 by 10×. The standard AI chip comparison misses most of what matters for inference.
We’re introducing the Inference Performance Index (IPI), a new measure that accounts for memory bandwidth and inter-chip communication alongside compute. These factors determine how many users or agents a chip can support simultaneously at a given output speed.
Scores in GB300 equivalents let you compare chips directly and estimate the total inference capacity of a data center, or even the world.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to US-China foundation model benchmarks and commercialization comparison and AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include NVIDIA, GB300, H100, benchmark.
Source attribution: curated from Epoch AI (@EpochAIResearch).
