SWE-bench 作者 Ofir Press 谈新基准该从 0% 起步
SWE-bench 作者 Ofir Press 分享基准设计心得:上线时最强模型应接近 0 分。他举了 SWE-bench 从 2% 到接近满分的历程,还提到 Cognition 的 Devin 是引爆点。做评测或想看懂榜单的开发者都该听听这段对话。
SWE-bench 共同作者 Ofir Press 在 Kaggle 的 Evaluated 播客中指出新基准构建者的两个常见错误:起点太容易,以及任务没人真正关心。
他建议基准发布当天最强模型得分应接近 0%,并选择易于解释、易于评判但极难执行的自然任务。SWE-bench 发布时最强模型只解决了 2% 的任务,前三四个月被认为几乎不可能完成,直到 Cognition 在 Devin 发布会上引用该基准后才引发团队竞逐。
如今模型在 SWE-bench Verified 上已能解决大多数任务。Ofir 认为下一代基准需要寻找当前完全无法完成的任务,设计时要"以超人标准思考"。
"If you're launching a benchmark that starts out at 50% accuracy, that objective has almost already been reached," says @OfirPress, co-author of SWE-bench.
On the Evaluated podcast, Kaggle's Gina Lee asked him about the mistakes new benchmark builders make:
1️⃣ Starting too easy. The best models should score around 0% on launch day, so there's room to climb. 2️⃣ Choosing tasks nobody cares about. Go with natural tasks, the ones we really want models to do well.
To avoid these mistakes, Ofir recommends tasks that are "easy to explain, easy to judge, but also extremely hard to execute on."
When SWE-bench launched, the best model solved 2% of its tasks, which were based on bugs from GitHub. For the first three or four months, Ofir says, people thought it was too hard, maybe impossible. Then Cognition cited SWE-bench in its Devin launch, and teams started competing. Today, models solve most tasks on SWE-bench Verified.
The next benchmarks must find even more difficult tasks. "We need north stars for things that are totally impossible right now," Ofir says. To build a strong benchmark today, "you need to think superhuman."
What would you build a 0% benchmark for?
This report is categorized under AITOP's "Industry" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include SWE-bench, Devin, Cognition, Kaggle.
Source attribution: curated from Kaggle (@kaggle).