Alibaba and Cornell paper benchmarks verification designs for coding agents
The cost-efficiency split matters: a final critic is nearly free, while completion gates buy accuracy at much higher cost, so harness builders can mix checkpoints by budget. Teams building coding agents or agent harnesses can use these numbers to pick where verification belongs in their pipelines. The Terminal-Bench and SWE-bench Verified figures give concrete baselines for comparing designs against named frontier models.
Alibaba and Cornell researchers compared 17 verification designs across seven frontier coding models on Terminal-Bench and SWE-bench Verified. Verification checkpoints included initial instructions, reminders during runs, a completion gate, and an independent critic. A separate reviewer agent added 5.9 to 9.5 points on Terminal-Bench at 1.13 to 1.25x the baseline cost. The independent critic offered the best accuracy per dollar, adding 3.9 to 6.6 points on SWE-bench Verified at almost no extra cost. Multiple review rounds added up to 3.5 more points on Terminal-Bench.
Great paper from Alibaba and Cornell on verification in coding-agent harnesses.
(bookmark it)
If you build agent harnesses, this one is worth your time.
They find that a separate reviewer agent adds 5.9 to 9.5 points on Terminal-Bench at 1.13 to 1.25x the baseline cost.
They compared 17 verification designs across seven frontier coding models on Terminal-Bench and SWE-bench Verified. The designs check the agent's work at four points: the initial instructions, reminders during the run, a gate before the agent can finish, and an independent critic at the end.
Instructions and reminders gave small, model-dependent gains. Completion gates gave larger gains, sometimes at much higher cost. The critic gave the best accuracy per dollar, adding 3.9 to 6.6 points on SWE-bench Verified at almost no extra cost. More review rounds added up to 3.5 more points on Terminal-Bench.
https://t.co/uYBKCDSd50
Chat with Paper: https://t.co/Y3G0M6WpPo
