arXiv 论文用注册式重放检验 Adam 中 epsilon 对低频词嵌入行的实际作用
Replaying the Same Gradients at a Different $\varepsilon$: A Registered Test of the Fixed-Trajectory Account of Adam's $\varepsilon$ for One Rare Row
论文用一种很较真的注册式重放方法，证明 Adam 里调大 epsilon 不只是简单缩小低频词的步长，实验设计本身挺值得学。
这篇 arXiv 论文（编号 2610.38169）检验了 Adam 优化器中 epsilon 对低频 token 嵌入行的影响是否只是简单的算术缩放。作者在训练一个小型因果 Transformer 时，对一条探测行用不同 epsilon 重放同一梯度轨迹。结果显示：当 epsilon 提高 4 个数量级时，3 个注册种子全部判定该解释不成立，探测行保留的位移是预测值的 8.9–11.4 倍（主采样负例头），全 softmax 头下为 5.0–7.7 倍。在 3 个数量级时测试结果未越界，作者明确声明结论仅限单一模型族的一条嵌入行。
Replaying the Same Gradients at a Different $\varepsilon$: A Registered Test of the Fixed-Trajectory Account of Adam's $\varepsilon$ for One Rare Row
Adam divides every update by the square root of a running average of squared gradients plus a small constant, epsilon. For an output-embedding row whose token is rare, which we call the probe row, raising epsilon can be read as shrinking that row's steps by arithmetic alone. We test that reading directly. While a training run proceeds, we also divide the probe row's update by a different epsilon, using the run's own moment estimates; this replay is exact arithmetic on the gradients the run actually saw, so it holds the run's trajectory fixed. A rule for when the replay counts as wrong was written down before the runs. We train a small causal transformer in a standard configuration. When epsilon rises by four orders of magnitude, the replay counts as wrong on each of the three seeds registered for the test: the probe row keeps 8.9--11.4 times more of its displacement than predicted with the main, sampled-negative head, and 5.0--7.7 times more with a full-softmax head scored as a second instrument. At three orders of magnitude, tested with the main head only, it does not, by narrow margins; a further two seeds, registered after this result had been committed and scored on their own, follow the same pattern. In separate real runs, raising epsilon on the probe row alone is enough to reproduce the row's change and raising it on every other parameter alone also moves the row part of the way --- readings about the runs that do not rescue the replay; scaling the row's learning rate instead, with the denominator untouched, is mispredicted by the replay in the same direction. This is one embedding row of one model family on one corpus; we make no claim about rates, about rows in general (other rows were read only as a panel on one trajectory) or about how the effect changes with model size.