Jason Wei 统计 2026 年前沿实验室基准报告频率分布
Jason Wei 把各家实验室都在报哪些基准拉了张清单。能看出 Scale AI 造基准的话语权有多大，OpenAI 又是最大的基准供应商。做模型评测或关注实验室动态的人都值得对照一下，看看自己追的榜单是谁出的。
前 OpenAI 研究员 Jason Wei 发布了一份"benchmark benchmark"，统计了 2026 年 12 家前沿实验室报告中使用的基准频率分布。
HLE、SWE-Bench Pro、DeepSWE 和 Terminal-Bench 实现了全量采用，12 家实验室全部报告这四项。其中 Scale AI 背后参与了四个中的两个（SWE-Bench Pro 和与 CAIS 合作的 HLE），另外还有排名第 9 的 MCP-Atlas。
OpenAI 是实验室中最活跃的基准生产者，出品了 SWE-bench Verified、BrowseComp、GDPval；Anthropic 只有 BioMysteryBench 一项上榜，且几乎无人使用。
学术机构方面，Princeton（SWE-bench、CharXiv）、Stanford（Terminal-Bench）、Berkeley（CyberGym、Agents' Last Exam）和 NYU（GPQA）表现突出。
DeepSWE 从 6 月首次出现到被全部 12 家实验室报告，一年内 Terminal-Bench 更迭代了 2.0 到 4.
0 四个版本。
Created a "benchmark benchmark": a frequency distribution of which benchmarks were reported by the frontier labs in 2026: https://t.co/GswjxqWz1c
Some observations: - HLE, SWE-Bench Pro, DeepSWE, and Terminal-Bench saw universal adoption - Scale AI is behind two of those four (SWE-Bench Pro, HLE w/ CAIS), plus MCP-Atlas (#9 benchmark) - Of the frontier labs, OpenAI is the most prolific benchmark producer (SWE-bench Verified, BrowseComp, GDPval); people don't seem to mind evaluating on a benchmark made by a competitor - Anthropic has just one on the chart (BioMysteryBench) that basically nobody uses 🤔 - In academia, strong showings from Princeton (SWE-bench, CharXiv), Stanford (Terminal-Bench), Berkeley (CyberGym, Agents' Last Exam), and NYU (GPQA) - DeepSWE saw incredible adoption, from first appearing in June to all 12 labs reporting it - Labs cycled through four versions of Terminal-Bench (2.0 → 4.0) in a single year
This report is categorized under AITOP's "Industry" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include SWE-bench, Terminal-Bench, HLE, DeepSWE.
Source attribution: curated from Jason Wei (@_jasonwei).
