如果你在做低比特量化，ReRound用扩散模型解决RTN的中点模糊，3/4比特下比RTN更准，还不用校准数据，值得看看。
ReRound是一种后训练量化方法，针对标准RTN在量化接近区间中点的权重时固有的中点模糊性问题。它训练一个条件扩散模型生成低比特权重的连续重建，作为确定舍入方向的引导信号。ReRound引入容差度量，对中点附近权重用扩散重建，对边界附近权重用RTN，并通过扫描容差参数选择与原始全精度权重奇异值最匹配的候选。在3比特和4比特量化下，ReRound在小型LLM上持续优于RTN，且精度超过多种无校准方法，与依赖校准的方法相当，推理时无额外开销。该方法适用于LLM之外的AI模型，论文聚焦于小型LLM。
ReRound: Reconstructive Rounding to Resolve Midpoint Ambiguity in Calibration-Free LLM Quantization
ReRound (Reconstructive Rounding) is a post-training quantization method that addresses the midpoint ambiguity inherent in standard round-to-nearest (RTN) schemes when quantizing weights near the centers of quantization intervals. Starting from a pretrained LLM, ReRound trains a conditional diffusion model to produce continuous reconstructions of low-bit weights for the LLM. These reconstructed weights act as a guidance signal to disambiguate the rounding direction of weights located close to interval midpoints. To integrate this reconstruction-guided rounding with conventional RTN, ReRound introduces a tolerance metric measuring how far the quantized weight (not the final quantized integer) is away from the midpoint: quantized weights within a tolerance region around midpoints are quantized using diffusion-based reconstructions, whereas weights closer to quantization boundaries are quantized with RTN. By sweeping the tolerance parameter, ReRound generates multiple candidate quantized integer weight matrices and selects the de-quantized weight matrix candidate whose leading singular values most closely match those of the original full-precision weights. This selected candidate determines the tolerance parameter ReRound uses. ReRound is particularly effective for smaller LLMs. Across a range of such models, it consistently outperforms standard RTN for 3-bit and 4-bit weight quantization. ReRound achieves superior accuracy compared to an extensive set of calibration-free methods, remains competitive with calibration-dependent approaches, and operates entirely offline, introducing no additional overhead during low-bit inference. The ReRound strategy represents a new approach for low-bit quantization. The method applies to AI models beyond LLMs. This paper focuses on its applications to small LLMs.