Hamel Husain 回应质疑：为什么要公开批评 AI 基准测试的缺陷
做模型选型的朋友看看这篇，Hamel 教你怎么用半小时翻 trace 就能发现基准测试里的坑，别只看跑分。
Hamel Husain 针对 three 类常见反驳逐条回应：有人认为基准作者已知缺陷正在修复，有人批评者应直接贡献而非挑刺，还有人认为非黑即白的判定过于绝对。他指出这些基准影响着数百万美元的 token 支出决策，大量用户只看头条分数而不检查 failing tasks 的任务真实性、harness 配置、LLM judge 打分是否对齐 ground truth。他建议查看失败任务的 trace，往往第一条就能发现严重问题，最终应投入建设自己的私有 eval。
I've seen a lot of pushback to this, with the following complaints:
1. "The benchmark owners already know about the flaws, and are working on it." 2. "Why are they spending time critiquing and not contributing to the benchmark" OR "critiquing known issues are not helpful". 3. "The flawed verdict is too binary"
I find this pushback to be misguided. These benchmarks are used to move millions of dollars of token spend, and many people blindly trust how well these benchmarks generalize to their own products, and don't look any further than the headline numbers.
For example, whenever a new model drops, you'll find thousands of youtube videos analyzing the pareto frontier using one of these benchmarks as the primary quality metric without questioning the benchmark.
I think we would all be better served by inviting skepticism and encouraging people to develop an independent opinion of how well these benchmarks represent the tasks they care about. Concretely:
- For the failing tasks within the benchmark, are they: realistic, have a proper harness, graded properly, sufficiently disambiguated user intent? - Are models and API calls configured properly (ex: inconsistent context windows, reasoning levels). - Are the models equipped with a modern harness and minimal retrieval that is sufficiently realistic? - Does the score even make sense ex: is it just a LLM judge committee that produces a score on a 0-100 scale that isn't aligned with ground truth?
It's often the case you'll see many of these issues while looking at traces of a failing benchmark tasks, and it doesn't take long! Often the **very first trace you look at** has a serious issue.
I also don't think there is an issue with verified vs. flawed verdict. It's good to have opinions. If you look through enough of what's behind their opinions and you find that you usually agree with their verdict, then they have earned your trust, the same way that a benchmark can lose your trust.
I think this eval of benchmarks is a red pill that motivates investing in your own private evals.
