xeophon 揭露 AutomationBench 与 OfficeQA Pro v2 基准评分缺陷
有人扒出两个流行基准的评分器有硬伤，修好后分数直接打满，做评测前真该自己看看数据。
xeophon 发现 AutomationBench 强制模型用散文而非 JSON Schema 描述请求体，修复评分器后该基准接近饱和。OfficeQA Pro v2 的评分器使用 regex 和 Python 抽取答案，拒绝了大量正确回答，且模型疑似通过联网从 HuggingFace 拿到了答案原文。两处问题修复后基准同样接近饱和。Hamel Husain 借此提醒应审视原始数据并依赖私有评测。
Amazing BenchmarkGate [1] reveal by @xeophon where he found serious (even comical) problems in popular benchmarks:
AutomationBench: "[forces the models to] describe their request bodies in prose rather than in JSON Schema" 😅
He also fixed a number issues with the grader, which when combined with the above result in a near saturation of the benchmark!
OfficeQA Pro v2: the grader uses a regex and Python-based extraction, which results in rejecting many valid answers. Additionally, there is evidence of models using web access to retrieve the answer key from HuggingFace (receipts are in the post).
With these and other fixes, this again results in near saturation of the benchmark!
It's very impt to look at data and not trust metrics from third parties. Better yet, you should rely on your own private evals.
[1] BenchmarkGate: the gap between the [un]trustworthiness of benchmarks and the extent to which people rely on them.
