这篇论文用 QuoteBench 拆穿了只看匹配分数的假象，原来 GPT-5.6-sol 的分数是负 64 和正 60 抵消出来的，部署配置一变排名就翻，评估编程智能体前该看看。
QuoteBench 用 56 个单次任务、14 个事故来源族，验证 LLM 编程智能体在生成命令与执行传输之间的边界。同一回复经新增解析器后，成功率下降 55.4 到 73.2 个百分点；六种配置在披露边界后恢复 30.4 到 60.7 个百分点，另两种恢复为零或负值。GPT-5.6-sol 的匹配分数仅 -3.6 点，但实际隐藏了 -64.3 点损坏和 +60.7 点补偿。部署配置会重排模型顺序，26 对可比模型中有一对明确反转，另有四对处于单任务边缘。论文建议评估命令型智能体时报告模型配置、生成合同、执行路径、操作点和最终状态验证器，而非只给匹配分数。
QuoteBench: How Matched Scores Can Hide Command-Path Failures
LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.