Papers

TempQuant:为量化模型自适应调节温度与 top-p 采样参数

TempQuant: Learning Adaptive Temperatures for Quantized Language Models

Why we picked it

如果你在跑量化模型还照抄全精度的温度设置,这篇论文会改变你的做法。3-bit Qwen3-4B 只调采样参数就把 MATH 从 34% 提到 51%,不用标注数据。做低精度部署和端侧推理的工程师值得实际测一下。

arXiv 论文研究了量化与采样参数的交互:量化会系统性地使模型输出分布“升温”,且温度对量化模型性能的影响远大于对全精度模型。论文提出 TempQuant,通过学习采样温度和 top-p,让量化模型的下一个词概率分布更接近全精度模型。方法提供两种粒度:一是每个请求温度对应一个全局设置,无需标签即可逼近用标注答案调参的效果;二是逐 token 设置温度与 top-p——研究发现排名在量化后翻转的 token 倾向需要升温,排名保持不变的 token 倾向需要降温。实验显示,在 3-bit Qwen3-4B 上,全局 TempQuant 将 MATH 准确率从 Qwen 官方量化版本推荐配置下的 34% 提升到 51%。

Body

这项研究的直接意义是纠正了一个常见做法:模型卡通常建议量化模型沿用全精度版本的采样设置,但论文证明换成针对性设置可以显著提高准确率。当量化严重损害精度时,TempQuant 能恢复大部分损失;退化较轻时也能让行为更贴近全精度模型。对在低成本硬件上部署低精度模型的开发者来说,这是零标注成本的调参方案。

相比量化算法本身的改进(如 GPTQ、AWQ),TempQuant 不动权重、只调解码参数,工程成本更低,可作为现有量化管线的即插即用补充。逐 token 粒度版本还揭示了量化失真的结构规律,为后续研究提供了方向。

Original · arXiv cs.LG

TempQuant: Learning Adaptive Temperatures for Quantized Language Models

Large language models are increasingly deployed at low precision, yet quantization can degrade their performance. We study how quantization interacts with decoding settings such as the sampling temperature and top-p by examining how it distorts the model's next-token probabilities. Measuring entropy during generation across deployed temperatures, we find that quantization warms the model on average, and that temperature matters much more for the performance of a quantized model than for its full-precision counterpart. Because quantization and temperature both shape the next-token distribution, we investigate how much the sampling settings can undo what quantization changes, through our method TempQuant, which learns the quantized model's temperature and top-p to bring its next-token probabilities closer to the full-precision model's. We learn these settings at two granularities. The first is a single global setting per requested temperature, which finds nearly the same temperatures that tuning on labeled answers would, without any labels. Model cards typically recommend sampling a quantized model with the same settings as its full-precision counterpart, while we find that different settings can substantially improve its accuracy. On 3-bit Qwen3-4B, for example, global TempQuant raises MATH accuracy from 34% under the configuration recommended for Qwen's quantized releases to 51%. The second sets the temperature and top-p for every token, motivated by our finding that tokens whose rankings flip under quantization tend to need warming, while tokens whose rankings are preserved tend to need cooling. When quantization degrades accuracy substantially, TempQuant recovers much of it, and when degradation is lighter, it still brings the quantized model's behavior closer to the full-precision model's.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include TempQuant, Qwen3, quantization, MATH.

Source attribution: curated from arXiv cs.LG (@Mukund Athmaram, Liang Ma, Manoj Rao).