Double descent explained via statistical mechanics
Double descent is the principle of least action
For those interested in the theoretical foundations of machine learning, this paper provides a new perspective on a well-known phenomenon using statistical mechanics concepts.
This academic paper analyzes the double descent phenomenon in machine learning models, where test error first decreases, then increases, and decreases again as model parameters increase. It uses statistical mechanics to explain this by modeling training as a particle moving over an energy landscape at a temperature T, with the Boltzmann distribution governing parameter visits. The paper argues that adding parameters lowers the effective temperature and increases weight regularization, reducing large parameter values.
Double descent is the principle of least action
The test error of a model plotted against its number of parameters $d$ falls, peaks when the model can just fit the training data, and falls again, exhibiting the double descent phenomenon. We explain the phenomenon with statistical mechanics. The training trajectory of a stochastic gradient-based method is a particle wandering over the energy landscape of the training loss at an induced temperature $T$, and a run that has equilibrated visits every parameter vector of a given training loss equally often, the fundamental postulate of statistical mechanics, with probability given by the Boltzmann distribution. Because training starts at an initial point and has only finite time to diffuse, it carries an effective weight decay, which makes every parameter a quadratic degree of freedom. The equipartition theorem then distributes the energy among the $d$ degrees of freedom in shares of $T/2$, so at a fixed training loss adding parameters lowers the temperature and drives the Boltzmann distribution toward the stationary path. Finally, adding parameters can only lower the $L^2$ norm of the stationary path, so a solution sampled at fixed loss is less likely to be large with increasing $d$, effectively increasing weight regularization.