CertiSum:让 MXFP4 训练在不同平台与并行布局下结果完全一致
CertiSum: Bitwise-Identical MXFP4 Training Across Platforms and Parallelism Layouts
跨厂商 GPU 和并行方式训练结果居然会不一样,这坑很多人踩过。这篇提出 CertiSum,让 4-bit MXFP4 训练逐位可复现,实测在 AMD 和 NVIDIA 卡上都一致,速度还比 BF16 快。做分布式训练和审计的朋友值得花时间看看。
CertiSum 是一种让 MXFP4(4-bit 微缩放格式)训练实现逐位一致(bitwise-identical)的方法,解决同一输入在不同并行布局和 GPU 厂商之间产生不同结果的问题。
其核心观察是:MXFP4 乘积的 FP32 原生累加在多数求和顺序下本身就是精确的。CertiSum 在量化阶段收集统计信息来认证矩阵乘法结果,对未通过认证的输出进行精确重算,在认证几乎不成立的场景改用精确 kernel。它保持量化块在不同布局下不变,并在注意力与权重梯度计算中合并不带舍入的局部和。
在 200 步 Qwen3-8B-Base 实验中,CertiSum 在 AMD MI350X 与 NVIDIA GB200 之间以及 MI350X 上数据、张量、上下文并行布局切换时逐位保留训练状态。在 MI350X 上对 FineWeb-Edu 的三个 8B–14B 模型测试,它比非精确 MXFP4 对应实现增加 11.8–60.5% 的单步延迟,同时相对 BF16 最高加速 1.24×,相对确定性 BF16 最高加速 1.70×。
CertiSum: Bitwise-Identical MXFP4 Training Across Platforms and Parallelism Layouts
Training with MXFP4, a 4-bit microscaling format, is non-deterministic: identical inputs can produce different results across parallelism layouts and GPU vendors. These differences complicate debugging and training audits and can cause training and inference engines to disagree. We present CertiSum to make MXFP4 training bitwise invariant across runs, parallelism layouts, and GPU vendors. Its key insight is that native FP32 accumulation of MXFP4 products is often already exact for every summation order. CertiSum collects statistics during quantization to certify native matrix multiplication results, recomputes uncertified outputs exactly, and uses exact kernels directly where certification rarely succeeds. CertiSum keeps quantization blocks unchanged across layouts and merges partial sums without rounding, including in attention and weight-gradient computation. In 200-step Qwen3-8B-Base experiments, CertiSum preserves training states bit for bit across AMD MI350X and NVIDIA GB200 and across changes in data, tensor, and context parallelism on MI350X. Across three 8B--14B models on FineWeb-Edu using MI350X, it adds 11.8--60.5% to training-step latency relative to its non-exact MXFP4 counterpart at the larger evaluated input shapes, with speedups of up to $1.24\times$ over BF16 and $1.70\times$ over deterministic BF16.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include MXFP4, low-precision training, reproducibility, Qwen3.
Source attribution: curated from arXiv cs.LG (@Steve Rhyner, Yubo Gao, Stefan de Lasa, Sankeerth Durvasula, Nandita Vijaykumar). Cross-source corroboration: 4 independent sources have covered this development (including TestingCatalog, CoreWeave, The Information), confirming industry-wide signal strength.