FP4 预训练让模型学罕见事实多花约 1.2 倍曝光，验证损失却看不出
What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts
这篇实验做得很细：FP4 训练损失几乎一样，但罕见事实要多喂 1.2 到 1.5 倍数据才学会。做量化训练或挑低精度方案的人该看看，尤其里面 Adam 噪声那组对照实验能帮你避坑。
arXiv 论文（编号 2611.12832）用模拟 4-bit 矩阵乘法从零训练 transformer，在合成传记数据上控制每条事实的曝光次数，检验低精度预训练是否真的只影响损失。结果显示：NVFP4 需要 BF16 的 1.32 倍到 1.52 倍曝光才能匹配事实召回率，但两者验证损失仅相差 0.5% 到 2%；换用 BF16 矩阵乘法评估时差距降到 1.26 倍至 1.30 倍，主读数下匹配损失时的曝光成本约 1.2 倍。
机制上，零均值梯度噪声可以复现只量化反向传播时的成本，而这种量化几乎不改变损失。受控实验发现，进入 Adam 二阶矩估计的噪声相当于调低学习率、成本会消退；进入更新本身的噪声会降低信噪比，约三分之二的成本一直延续到训练结束。单纯调高学习率无法弥补变小的平均步长，对噪声和 NVFP4 都不奏效。
作者还证明用 BF16 检查点的梯度统计即可对各种量化格式、部分量化和噪声控制方案的成本排序，Spearman 相关系数达 0.96，无需实际拟合。结论是评估低精度训练方案时应把罕见事实习得和损失一起测。
What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts
Low-precision pretraining is usually judged by validation loss. We ask whether matching loss also means learning rare facts as efficiently. We train transformers from scratch with emulated 4-bit matrix multiplications on synthetic biographies with controlled exposure counts, on their own and mixed into web text. NVFP4 needs 1.32$\times$ and 1.52$\times$ as many exposures as BF16 to match its recall while validation loss differs by only 0.5% and 2%, and 1.26$\times$ and 1.30$\times$ when both models are evaluated with BF16 matrix multiplications. Under our primary readout, the exposure cost is about 1.2$\times$ at matched loss. Zero-mean gradient noise reproduces the cost of quantizing only the backward pass, which barely moves the loss. In a controlled experiment at twice the NVFP4 error, noise fed only to Adam's second-moment estimate acts like a lower learning rate and its cost fades, whereas noise in the update lowers its signal-to-noise ratio and carries about two thirds of the cost through the end of training. Raising the learning rate to offset the smaller average step does not recover the cost, either for this noise or for NVFP4. Gradient statistics of BF16 checkpoints rank the costs of formats, partial quantizations and noise controls without fitting (Spearman 0.96). We suggest measuring rare-fact acquisition alongside loss when evaluating low-precision recipes.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include FP4, NVFP4, BF16, 低精度训练.
Source attribution: curated from arXiv cs.LG (@Chang Lu, Mengjiao Zhang).