Stanford AI Lab 提出 Self-Play 预训练：零真实数据也能训练语言模型
斯坦福团队做了个疯狂实验：不用任何真实数据，靠两个模型自博弈生成训练数据，结果还真学会了上下文学习。
Stanford AI Lab 的 Michael Y. Li 等人提出 Self-Play Pretraining with Zero Data 方法。两个模型均从随机初始化开始：generator 为通用图灵机生成程序，learner 在其输出上训练，全程不使用任何真实数据。结果显示零样本验证损失在图像、文本、音频和旋律等自然数据上随自博弈计算量可预测地下降，且 learner 发展出 in-context learning 能力。作者称这是一项概念验证工作。
What would pretraining with zero real data look like? Can a randomly initialized model learn to generate all of its training data entirely through self-play? Find out below! :) Michael Y. Li @michaelyli_ Can an LM, starting from random init (!!), learn to generate all of its pretraining data? Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator proposes programs for a universal Turing machine and a learner trains on their outputs. We never train on any real data, but see predictable scaling on natural datasets: zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute. And the learner develops in-context learning capabilities. A fun proof-of-concept, co-led with @AdityaCowsik and @KfirDolev and co-authors @gbruno_dl , @ANourya @noahdgoodman , and @YoavLevine . 🔗 View Quoted Tweet 💬 8 🔄 16 ❤️ 182 👀 18444 📊 35 ⚡