Precision Choices Break Greedy Decoding Determinism in LLMs
Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference
Anyone chasing reproducible LLM outputs — evaluation harness authors, serving teams and quantisation engineers — should treat BF16 vs FP16 as a real source of run-to-run variance, not an implementation detail. The paper offers a low-overhead fix (threshold-triggered FP32 LM head recomputation) and maps when it fails, including FP8 end-to-end serving. The margin-based analysis also gives practitioners a quick diagnostic for whether their own deployments are at risk.
A research paper shows that greedy decoding in LLMs is not precision-invariant: the same model, prompt and hardware produce different outputs under BF16 versus FP16. Across six models (1.1B-7B parameters, four families) and three benchmarks, 49-100% of prompts diverge, with a single token flip cascading into trajectory-level divergence. The authors find divergence depends mainly on the top-two logit margin at the LM head, and their error-propagation analysis yields five testable predictions that all matched experiments. A selective FP32 LM head recomputation intervention improves exact agreement by 22-36 pp on A10G at under 4% latency overhead for batch size <=4, though the benefit vanishes at batch size >=8 or under end-to-end FP8.
Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference
Greedy decoding from large language models is commonly treated as deterministic. We show it is not precision-invariant: the same model, prompt, and decoding algorithm produce different outputs in BF16 versus FP16 on identical hardware. Across our evaluations of six models (1.1B-7B parameters, four families; divergence additionally characterised at 12B) and three benchmarks, 49-100\% of prompts diverge; a single token flip often cascades into trajectory-level divergence. We develop an empirical error-propagation analysis and find that 22 layers of accumulated body error do not distinguish flipping from non-flipping steps; the outcome depends primarily on the top-two logit margin at the LM head relative to the directional perturbation between the top-two candidates. The analysis makes five testable predictions about intervention outcomes, including that applying more FP32 compute (broader scope) makes agreement worse. The experiments match all five predictions. The best-performing low-overhead intervention we evaluate, selective FP32 LM head recomputation, triggered only when the margin falls below a threshold, delivers +22-36 pp exact agreement on A10G (+12-21 pp on L4 and A100) at less than 4\% latency overhead in low-batch (batch size <=4) single-stream inference. We map the applicability boundary across six models and four batch sizes, and hypothesise that training-time precision stability is a determining factor. The method is a partial mitigation rather than a universal determinism guarantee: its benefit vanishes when body-originated error dominates, including at batch size >=8 and under end-to-end FP8 in our tests.
