Meta paper shows byte-level models outperform token models at scale

The study tested distilled 1B models trained on up to 1 trillion bytes, finding that byte models reach a higher performance ceiling.

The fitted scaling laws predict the End-Of-Token model will be 4% ahead of the distilled token model at large scales.

Byte models also match token models with one-sixth the training data, and a 256-entry vocabulary reduces teacher-logit storage to about a fifth.

Banger paper from Meta. This work shows that byte-level models start out behind token models and then pass them as compute grows. They show this for distilled 1B models trained on up to 1 trillion bytes. To distill a byte student from a token teacher, they convert the teacher's token logits into byte logits, either approximately (Marginalize-It) or exactly (End-Of-Token). Token models lead at low compute but plateau. Byte models reach a higher ceiling, and the fitted scaling laws predict the End-Of-Token model ends up to 4% ahead of the distilled token model. The byte models also match the distilled token model with one-sixth of the training data, and a 256-entry vocabulary cuts teacher-logit storage to about a fifth. Paper: arxiv.org/abs/2609.12303 Chat with Paper: academy.dair.ai/papers/breakin…📊 12 ⚡

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Meta, byte-level models, token models, scaling laws.

Source attribution: curated from elvis (@omarsar0). Cross-source corroboration: 2 independent sources have covered this development (including 量子位), confirming industry-wide signal strength.