Google DeepMind 工程师谈智能体评测:通过测试不等于做对了事
做智能体评测的都该看看这个演讲:DeepMind 的人教你像严格的代码 reviewer 一样给 agent 打分,正确性之外还要看风格、可维护性,还要防刷分。如果你的 agent 总是过测试但产出没法用,这套评分思路能直接拿来改你的 eval 流程。
Google DeepMind 的 melvinjohnsonp 在 AI Engineer Code Summit 演讲中提出,智能体即使通过全部测试用例,仍然可能偏离任务的真实意图。
他主张用类似严格代码审查者的方式来给智能体打分,评测维度覆盖正确性、代码风格和可维护性三项。演讲还强调要对钻空子行为施加惩罚,防止智能体通过投机手段刷高评测分数。
这套评分框架针对的是当前智能体评测中只看基准通过率的常见做法。
An agent can pass the tests and still miss the point. @melvinjohnsonp of @GoogleDeepMind on grading agents like a strict reviewer: correctness, style, maintainability, and penalties for gaming the system. From AI Engineer Code Summit.
This report is categorized under AITOP's "How-to" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include agents, Google DeepMind, benchmarks, AI Engineer Code Summit.
Source attribution: curated from AI Engineer (@aiDotEngineer).