跑大模型评测总烧钱？这个框架能省下最多97%的采样次数，结论还一样，做评测的人值得看看。
arxiv 论文提出 optstop 框架，将大模型评测视为序贯测量问题，只对不确定性高的样本继续采样。该框架基于分层贝叶斯推断，支持二值、有序和连续结果，无需校准题库。在 200 项、10 轮次的示例评测中，optstop 在九种验证设置下减少了 57% 到 97% 的计划试验，整体结论与完整评测一致。研究表明评测算力可按不确定性分配，而非固定重复次数。
Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations
LLM evaluations often use fixed sampling budgets, testing every item the same number of times even after estimates are precise. We introduce optstop, a precision-based adaptive stopping framework that treats evaluation as a sequential measurement problem: keep sampling where uncertainty remains high, and stop where estimates are precise or stable enough. The framework builds on hierarchical Bayesian inference, supports binary, ordinal, and continuous outcomes, and keeps every benchmark item eligible for sampling, without requiring a calibrated item bank. It runs live or retrospectively, and includes a safeguard that samples more cautiously as measured performance approaches zero, where rare successes matter most. In an illustrative 200-item, 10-epoch evaluation, it removes 57%-97% of planned trials across nine validation settings, with overall conclusions equivalent to the full run. These results show that LLM evaluation compute can be allocated by uncertainty rather than by fixed repetition counts, with the magnitude of savings depending on evaluation design.