Stanford AI Lab tests pretraining with zero real data via self-play
The result suggests that data synthesis by self-play alone can reproduce scaling behaviour normally attributed to real corpora, which challenges assumptions about what pretraining data must contain. Researchers studying synthetic data and training dynamics will want to test whether the predictable loss curves hold at larger compute budgets and on harder tasks.
Researchers at Stanford AI Lab introduced Self-Play Pretraining with Zero Data, an experiment where two models start from random initialisation: a generator proposes programs for a universal Turing machine, and a learner trains on the generator's outputs.
No real data is used anywhere in training.Despite this, zero-shot validation loss on images, text, audio and melodies decreases predictably with self-play compute, and the learner develops in-context learning capabilities.
The project, co-led by Michael Y.Li, Aditya Cowsik and Kfir Dolev with co-authors including Noah D.
Goodman, is described as a proof-of-concept.
What would pretraining with zero real data look like? Can a randomly initialized model learn to generate all of its training data entirely through self-play? Find out below! :) Michael Y. Li @michaelyli_ Can an LM, starting from random init (!!), learn to generate all of its pretraining data? Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator proposes programs for a universal Turing machine and a learner trains on their outputs. We never train on any real data, but see predictable scaling on natural datasets: zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute. And the learner develops in-context learning capabilities. A fun proof-of-concept, co-led with @AdityaCowsik and @KfirDolev and co-authors @gbruno_dl , @ANourya @noahdgoodman , and @YoavLevine . 🔗 View Quoted Tweet📊 35 ⚡
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Stanford AI Lab, synthetic data, self-play, pretraining.
Source attribution: curated from Stanford AI Lab (@StanfordAILab).
