Stanford paper finds bits-per-byte is not tokenizer-agnostic
BPB has been treated as a neutral yardstick in scaling papers, so correlations this weak undermine cross-tokenizer comparisons that many labs rely on. Practitioners choosing a vocabulary size should benchmark task accuracy on the same text rather than trusting a single loss-based score. The finding matters most to researchers running tokenizer ablations or citing BPB in model cards.
A Stanford paper shows that bits-per-byte (BPB), the metric commonly used to compare models with different tokenizers, is not truly tokenizer-agnostic. Untrained models guessing uniformly still received different BPB scores depending on the tokenizer, with English text scoring best near a 24K vocabulary while accuracy stayed at chance. On trained models sharing one recipe but using different tokenizers, correlations between BPB and task scores ranged only from +0.02 to −0.30. The authors also rechecked a published claim that another supposedly tokenizer-neutral metric agrees with BPB.
New Stanford paper finds that no metric can be tokenizer-agnostic in every sense, so anyone comparing tokenizers should not trust a single score and should also check whether the model answers correctly.
To compare models with different tokenizers, researchers usually report bits-per-byte, or BPB, which divides the loss by bytes instead of tokens. Many papers call it tokenizer-agnostic.
They scored untrained models that guess uniformly, plus released models that share one recipe but use different tokenizers. They also rechecked a published claim that another supposedly tokenizer-neutral metric agrees with BPB.
Models that learned nothing still got different BPB by tokenizer, with English text scoring best near a 24K vocabulary, while accuracy stayed at chance. On models trained with different tokenizers, correlations between BPB and task scores ran only from +0.02 to −0.30.
When comparing tokenizers, don't treat lower BPB as proof of a better model, and check task accuracy on the same text before picking a vocabulary.
