论文精选

StateM 运行时实现 Terminal-Bench 2.1 95.3% 原始准确率

StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling

精选理由

StateM 不改模型权重,光靠升级执行系统就把 GPT-5.6 在终端测试的准确率拉到 95.3%,成本只要 15 美元,值得看看它的思路。

AI 摘要

StateM 是一个不修改模型权重的 agent 运行时,通过状态管理和可恢复流程提升执行系统。在 Terminal-Bench 2.1 上,StateM 将 GPT-5.5 xhigh 从 83.1% 提到 92.1%,GPT-5.6 Sol xhigh 达到 95.3% 原始准确率,覆盖 445 次试验和全部 89 个任务。同一套 runbook 让 GPT-5.6 Luna 从 76.7% 升至 85.4%,超过 Sol xhigh 参考值 84.9%。用不到 38 美元的适配,DeepSeek-V4 Flash 从 82.7% 提高到 88.1%,在 88 任务核心集上到 89.1%,最终 API 花费约 15 美元,对比 GPT 参考的 574.68 美元。代码已开源在 github.com/henryqin1997/statem。

原文 · arXiv: DeepSeek

StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling

Long-horizon agents can fail even when their underlying models can solve the constituent steps. They may lose track of mutable state, fail to reactivate lessons from earlier executions, skip known procedures, or stop prematurely. We bet on harness scaling to improve the execution system around an agent without changing its model weights. We introduce StateM, an agent-native runtime that organizes execution around durable states, phase-local context, checked transitions, recoverable runbooks, and versioned procedural practices that agents and users can inspect together. On Terminal-Bench 2.1, StateM raises GPT-5.5 xhigh to 92.1\%, versus 83.1\% reference and GPT-5.6 Sol Ultra at 91.9\%. The runbook transfers unchanged to GPT-5.6. With GPT-5.6 Sol xhigh, StateM reaches 95.3\% raw accuracy across 445 trials and succeeds on all 89 tasks at least once. The frozen profile raises GPT-5.6 Luna from 76.7 to 85.4\%, above the 84.9\% Sol xhigh reference. Using the same runtime, runbook structure, and golden rules, less than \$38 of adaptation raises DeepSeek-V4 Flash from 82.7 to 88.1\% under standard timeouts and to 89.1\% on an 88-task common core. Extending only the remaining latency-sensitive task matches the reported 88.8\% GPT-5.6 Sol max result. Final-score API usage is about \$15 versus \$574.68 for the GPT reference; total DeepSeek expenditure is \$52.22. On BusinessBench, family-specific runbooks built on development sets yield held-out gains of 0.55 macro and 1.34 micro points; two mechanism-matched families improve by 10.04 points. Concrete rules generalize when tasks share execution structure, while the control methodology applies broadly. StateM turns selected postmortem findings into persistent, executable preconditions and practices, making learned controls explicit and enforceable through stateful controls. Code at github.com/henryqin1997/statem.