QAM 提出二次精度检查点合并方法，误差达 O(h^3) 最优界
QAM: Quadratic-Accurate Checkpoint Merging via Sequential Consistency
一篇理论味很浓的 checkpoint 合并论文：证明信息论下界，给出达到 O(h^3) 最优误差的合并系数，还在 SmolLM3-3B 上做了实测对比。
论文 QAM 研究如何用训练轨迹上保存的 checkpoint 精确重构串行训练的终点。在局部转移模型下，两个 checkpoint 索引矩条件刻画了所有与串行参考二阶一致的凸合并方案，并证明了 GD 历史无法达到 o(h^3) 误差的信息论下界。
QAM 方法达到匹配的 O(h^3) 一致误差界，其显式系数在所有固定二次目标上精确匹配串行 GD 参考。在 SmolLM3-3B 和 OpenEuroLLM-Prelude-9B 两条 Adam checkpoint 轨迹、每模型 3 个窗口和 3 种配置、共 15 个任务的实验中，QAM 在长窗口场景下优于 Warmup-Stable and Merge（WSM），短窗口表现则好坏参半。
GSM8K 诊断还显示局部一致性并不完全决定下游成绩。
QAM: Quadratic-Accurate Checkpoint Merging via Sequential Consistency
Saved checkpoints record states along a training trajectory, but generally do not determine the updates at states that would be visited under a different schedule. We study how accurately these checkpoints can reconstruct the endpoint of a sequential reference with prescribed update strengths. Under a common local transition model, two checkpoint-index moment conditions characterize all convex merges that agree with this reference through second order. We then prove an information limit that for nondegenerate profiles, no algorithm using only a fixed-length gradient-descent (GD) history with step size $h$ can achieve $o(h^3)$ endpoint error uniformly over a fixed class of smooth, strongly convex losses. The lower bound follows from two losses with identical GD checkpoint histories but sequential reference endpoints separated by $Ω(h^3)$. \textbf{Quadratic-Accurate Merging} (QAM) achieves a matching uniform $O(h^3)$ endpoint error bound. Its explicit coefficients also define the unique profile-dependent merge that exactly matches the sequential GD reference across all fixed quadratic objectives. Across two public Adam checkpoint trajectories (SmolLM3-3B and OpenEuroLLM-Prelude-9B), three windows and three profiles per model, and 15 tasks, QAM shows mixed results for short windows and broader advantages over \textbf{Warmup-Stable and Merge} (WSM) for longer windows. Matched-moment GSM8K diagnostics further show that local consistency alone does not fully determine downstream scores. These results characterize the reconstruction limits of saved histories, provide a coefficient rule that attains the optimal rate, and assess its practical utility.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include checkpoint merging, SmolLM3, GSM8K, 微调.
Source attribution: curated from arXiv cs.AI (@Shihao Wang, Rui Kong, Xinran Chen, Hui Wu, Qipeng Qian, Jinman Zhao, Jiashu Zhao, Yuchen Li, Jimmy Huang, Dawei Yin).