Papers

微调能换来多少解码确定性?边际正则化抑制批处理导致的 token 翻转

How Much Determinism Can Fine-Tuning Buy? Margin Regularisation Against Batch-Induced Token Flips

Why we picked it

一篇挺有意思的研究:贪心解码其实不是确定性的,同一 prompt 换个 batch 就可能输出不同 token。作者用 logit 边际正则化在 GSM8K 上把发散率从 53.8% 压到 38.1%,代价 1-3 个准确率点。做需要可复现输出的服务或评测的同学可以看看。

arXiv 论文(编号 2611.19135)研究了零温度贪心解码在批处理下并非确定性的问题:同一 prompt 因与其它请求共享 batch 而产生不同 token。作者在六个模型家族、三种数值精度上测量这种扰动,发现扰动幅度与可表示精度严格对应(bfloat16 对 float16 为 8.00 倍,float16 对 float32 为 8192.0 倍,与尾数位数预测一致),且某些张量形状会意外实现批不变性,仅靠设定序列长度即可获得零不确定性。

Body

在此基础上,论文提出对 top-1 / top-2 logit 差距施加 hinge 惩罚,缩放到实测噪声分位数的两倍。在 GSM8K 上,token 翻转减少 35%,发散率从 53.8% 降至 38.1%,代价是 1.3–2.9 个准确率点,且随边际大小单调增加。由于 hinge 是软惩罚,幸存翻转的 logit 差距中位数(0.125)在各实验组中保持不变。

这一方案把确定性保障从推理阶段(如批不变 kernel,以吞吐量为代价)部分转移到训练阶段,为需要可复现输出的场景提供了训练时选项。论文还检验并否定了七个假设,其中一个关于单模型可行性的诊断在六个模型上成立、在第七个上被证伪。

Original · arXiv cs.AI

How Much Determinism Can Fine-Tuning Buy? Margin Regularisation Against Batch-Induced Token Flips

Greedy decoding at zero temperature is widely assumed to be deterministic, although the exact same prompt may yield different tokens depending on the number of other prompts sharing the batch for inference. The currently available solutions operate during inference, trading determinism for throughput. Can we train a model in such a way that fewer decoding steps would be susceptible? We characterize the perturbation empirically across six model families and three numerical precisions and observe two empirical facts about the measurement: the perturbation depends exactly on the representable precision (8.00x for bfloat16 vs float16, 8192.0x for float16 vs float32, in line with predictions based on mantissa bits) and the batch invariance happens accidentally at certain tensor shapes, allowing a practitioner to achieve zero nondeterminism merely by setting the sequence length. Then we apply hinge penalization of the model's top-1 / top-2 logit gap, scaled to twice the measured noise quantile. On GSM8K dataset it leads to a 35% reduction in token flips and divergence from 53.8% to 38.1%, at the cost of 1.3 - 2.9 accuracy points, increasing monotonically with the margin size. It does not affect the gap threshold sufficient for determinism: the logit gaps of surviving flips are unchanged (median 0.125 for all arms), as the hinge is a soft penalty and the largest hardware perturbation remains the same. We reject seven hypotheses, including a per-model viability diagnosis, holding on six models and falsified on the seventh.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include GSM8K, fine-tuning, numerical precision.

Source attribution: curated from arXiv cs.AI (@Charan Teja Nagisettygari, Siva Raja Sindiramutty).