PapersSelected

AI Token Value Scaling Laws Study

Why we picked it

This research reveals a critical threshold where AI contamination in training data becomes detrimental, challenging assumptions about data quantity versus quality. For developers and researchers working with language models, this provides concrete guidance on data curation strategies and highlights the growing importance of distinguishing between human and AI-generated text in training datasets.

Researchers pre-trained 800 language models to test how AI-generated text affects model performance.

Body

They found that for data-starved models, adding AI tokens initially lowers loss on human text but the benefit reverses with more tokens.

For models trained on substantial human text, AI tokens raise loss immediately while fresh human tokens continue to lower it.

The study developed a new scaling law that predicts AI token effects with 41% lower error than existing laws.

Original · Tanishq Abraham (论文推介)

How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text

"After applying FineWeb quality filtering, we find that 27.5% of tokens from June 2026 web data are labeled as AI-generated by Pangram, rising to 31.1% by August."

"How does AI text in the wild affect language model pretraining? To answer this question, we pretrain 800 language models, varying the ratio of added AI tokens to human tokens, and fit scaling laws to held-out losses on both human and AI-generated text. For data-starved models, adding AI tokens to pretraining data initially lowers loss on human text, but the benefit saturates as more are added and quickly *reverses* into harm. For models trained on high budgets of human text, AI tokens raise loss almost immediately, whil"e the same number of fresh human tokens keeps lowering it."

"We propose a new scaling law with separate benefit and harm terms that allows the value of an AI token to change sign while also reducing to Chinchilla in the absence of AI text. When fit on smaller models, our scaling law predicts the effect of AI text on held-out human-text loss for models up to 3.6x larger with 41% lower error than the best existing law over all AI ratios."

link: https://t.co/HqVuEu3DEw

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include AI tokens, scaling laws, language models, pretraining.

Source attribution: curated from Tanishq Abraham (论文推介) (@iScienceLuvr).