这篇论文戳破了「更多Token=更好模型」的直觉，做小模型训练或资源受限场景的开发者会看到训练轨迹比终点指标更关键，建议点开看看如何用间隔遥测避免白费算力。
本研究在固定计算预算下训练了一个4.26百万参数的Llama风格小模型，使用TinyStories语料库和CPU全精度训练，累计约2000万训练Token。通过重复测量设计，在21个间隔收集了验证损失、困惑度、波动性等指标。结果显示，验证损失从初始的8.3552迅速降至约400万Token时的2.7996，但最终回升至3.9010，困惑度也呈现先降后升的非单调模式。研究还发现了验证损失的反复回退现象，且未观察到稳定阶段。这表明在计算受限环境下，仅依赖最终指标评估模型效率可能掩盖训练过程中的不稳定性、退化及收益递减问题。
A Quantitative Experimental Repeated Measures Study of Training Dynamics in a Small Llama Style Language Model Under a Compute-Aware Token Budget
This study examines training dynamics in a small Llama-style language model trained under a fixed, compute-constrained token budget. Rather than evaluating efficiency solely through endpoint performance, the study uses a quantitative experimental repeated measures design to analyze how validation loss, validation perplexity, rolling volatility, backslide behavior, spike behavior, and between-seed variability change across token-based training intervals. Six independent training runs were conducted on a 4.26-million-parameter model using the TinyStories corpus, CPU-based full-precision training, and a target budget of approximately 20 million cumulative training tokens. Metrics were collected across 21 intervals, producing 126 seed-by-interval observations. Repeated measures ANOVA showed statistically significant interval effects for validation loss, validation perplexity, and rolling volatility. Descriptive trajectories revealed rapid early improvement followed by non-monotonic degradation during later training intervals. Mean validation loss decreased from 8.3552 at initialization to 2.7996 near 4 million tokens, but increased to 3.9010 by the final checkpoint. Validation perplexity followed the same pattern, falling sharply early in training before rising later. Derived telemetry further showed recurrent validation-loss backslides and no interval-summary evidence of a stable phase under the predefined criteria. These findings suggest that compute-aware language model evaluation should examine training trajectories rather than endpoint metrics alone. In constrained compute settings, additional token exposure may increase computational cost without producing proportional generalization gains, and interval-level telemetry can reveal instability, regression, and diminishing returns that final metrics may obscure.