这篇论文用具体数字告诉你，准确率分数会骗人：同样2048 token，Qwen比DeepSeek多出近10倍的无答案失败，评估时得分开看执行状态和得分。
语言模型基准将是否到达可评估状态和答案是否正确合并为一个准确率分数。新提出的两层评估框架分离了执行证据（终止、答案暴露、可解析性、完成长度）和正确性。在5个Qwen和DeepSeek配置的2550个输出上，2048 token限制下：Qwen MATH有49/450未终止，DeepSeek MATH有5/300未终止，ARC无未终止。相同300个DeepSeek MATH输出在8192 token下无缺失最终答案终止。覆盖审计的目标验证表明候选选择与聚合策略可显著改变比较准确率估计。
Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets
Language-model benchmarks collapse two distinct measurement questions into a single accuracy score: whether a response reached an evaluable state, and whether its answer was judged correct. We introduce a two-layer evaluation framework that separates scorer-independent execution evidence, including termination, answer exposure, parseability, and completion length, from scorer-dependent correctness. Across 2,550 outputs from five fixed Qwen and DeepSeek configurations on MATH and ARC-Challenge, matched 2,048-token limits produce sharply different execution mixtures: 49 of 450 Qwen MATH outputs terminate without a final answer, compared with 5 of 300 DeepSeek MATH outputs and none of the 750 ARC outputs. Among the same 300 DeepSeek MATH question-model pairs, no missing-final length termination is observed at 8,192 tokens. A coverage-audited targeted verification study further shows that candidate-selection and aggregation policies can substantially alter comparative accuracy estimates. These results demonstrate that accuracy conflates execution case mix with verification policy. Evaluations of test-time methods should therefore report pre-intervention execution states, verification coverage, and scorer provenance alongside accuracy.