Scaling Theory Links Multi-Epoch Pretraining Overfitting to Ensembling
A Scaling Theory of Multi-Epoch Pretraining: Late-Time Overfitting and Ensembling
As datasets saturate, labs will reuse data across epochs, and this work predicts when that overfits and how ensembling shifts the optimal stopping point. Practitioners planning pretraining or distillation budgets get concrete allocation rules rather than intuition. The FineWeb experiments make the theory testable against single-model scaling baselines.
An arXiv paper (2611.03590) studies how training epochs and ensemble size jointly affect generalization when pretraining data, rather than compute, is the binding constraint. Experiments on FineWeb show that logit averaging across an ensemble reduces late-training deterioration, delays optimal stopping, and can outperform wider or deeper single models at matched parameter-token budgets. The authors derive a dynamical mean-field theory (DMFT) for multi-epoch SGD in power-law random-feature ensembles, showing ensembling cuts the prediction variance that drives late-time test error growth. From this they derive compute-allocation laws, with ensemble size growing with compute at least as fast as per-member training time.
A Scaling Theory of Multi-Epoch Pretraining: Late-Time Overfitting and Ensembling
In the near future language model pretraining will become bound by total data rather than total compute. Such a regime requires revisiting scaling laws and scaling law theory. Motivated by prior works that have found utility in ensembling, we study how the number of training epochs and ensemble size jointly shape generalization when training data are fixed. In language-model experiments on FineWeb, logit averaging reduces late-training deterioration, delays optimal stopping, and can outperform wider or deeper single models at matched parameter-token budgets over a common training horizon. To analyze this interaction between data reuse and ensembling, we derive a dynamical mean-field theory (DMFT) for multi-epoch stochastic gradient descent in power-law random-feature ensembles, tracking error correlations across epochs and models. Ensembling reduces the prediction variance responsible for late-time growth in expected test error, changing the optimal stopping time. We use these stopping times to derive compute-allocation laws at fixed data and individual model sizes. Under this allocation, ensemble size grows with compute at least as fast as training time per member, while excess test loss above the limiting loss decreases as a power of compute.
