Particle解码免费估计序列级Rényi熵，自适应推理降低算力开销
Free Lunch Uncertainty: Sequence-Level Rényi Entropy from Particle Decoding for Adaptive LLM Reasoning
做推理部署的可以看看：SMC 解码顺手就能算出序列级熵，不用额外跑补全。论文在 MATH500 上验证过，两遍策略用少 34.2% 的粒子就能保住 64 粒子的准确率，算力紧张的团队直接受益。
一篇新论文提出从 Sequential Monte Carlo（SMC）power sampling 解码中零成本读出序列级 Rényi 熵（SRE）：由于 SMC 本身已估计 power 分布的归一化常数，序列级熵可在任意粒子预算下免费获得，无需像 self-consistency 那样额外生成大量补全。作者用 Rényi 链式法则将读数扩展为一族逐步熵，并证明阶数大于 1 的成员与每步最可能的 token 绑定（因此与 log-probability 相关），而 1/2 阶成员累积每步搜索努力的对数。
实验显示该 1/2 阶信号在 13 个设置中的 10 个里是最好的零成本选择性预测信号；在 MATH500 上其 AUROC 比 log-probability 高 0.022 到 0.044，且在全部四个模型和各长度四分位内保持一致，held-out 设置下也选出同一阶数。
在实际应用上，一个 8 粒子探针可识别哪些问题需要更多粒子，配合两遍策略在 MATH500 上用少 34.2% 的粒子匹配 64 粒子的准确率（三个种子平均）。这意味着已在使用 SMC/power sampling 的团队可以免费获得不确定性估计并显著削减推理算力，而 token 级熵和 log-probability 等现有零成本信号只覆盖局部信息。
Free Lunch Uncertainty: Sequence-Level Rényi Entropy from Particle Decoding for Adaptive LLM Reasoning
Quantifying uncertainty over an entire LLM completion is critical for selective prediction and adaptive compute. Zero-cost signals such as mean token entropy and log-probability are local to tokens or to one completion, and self-consistency needs many extra completions. We show that Sequential Monte Carlo (SMC) decoding for power sampling already estimates the normalizing constant of the power distribution, which gives the sequence-level R'enyi entropy (SRE) of the model's completion distribution at zero marginal cost, with a guarantee at every particle budget. A R'enyi chain rule extends this readout to a family of stepwise entropies of any order. We prove that orders above one are tied to the most likely token of each step, which explains why they track log-probability, while the order-1/2 member accumulates the log of per-step search effort. This member is the best zero-cost selective-prediction signal in 10 of 13 settings. On MATH500 its AUROC is 0.022 to 0.044 above log-probability with all four models and within length quartiles, and the same order is selected when each setting is held out. An 8-particle probe identifies the problems that benefit from more particles, and a two-pass policy matches 64-particle accuracy on MATH500 with 34.2% fewer particles over three seeds.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Rényi熵, Sequential Monte Carlo, MATH500, 不确定性量化.
Source attribution: curated from arXiv cs.LG (@Erfan Baghaei Potraghloo, Milad Heidari Sorkhehdizaj, Shahab Rahimirad, Aliasghar Pourghani, Seyedarmin Azizi, Souvik Kundu, Abolfazl Hashemi, Joseph Campbell, Vijay Gupta).