Scaling Theory Links Multi-Epoch Pretraining Overfitting to Ensembling
As datasets saturate, labs will reuse data across epochs, and this work predicts when that overfits and how ensembling shifts the optimal stopping point. Practitioners planning pretraining or distillation budgets get concrete allocation rules rather than intuition. The FineWeb experiments make the theory testable against single-model scaling baselines.