IndustrySelected

Robin Salimans 在 Bench Summit 2026 分享 AutomationBench 1.5 设计经验

Why we picked it

做评测或智能体的人可以听听这套思路:砍任务数换多次试验,测真实可靠性而不是单次运气。顶级模型发布时不到 10%,难度校准相当狠,想自建基准的团队基本能照着抄。

Kaggle 转发了 Robin Salimans 在 Bench Summit 2026 的演讲要点,回顾他在 Zapier 提出构建业务智能体基准的经历。Zapier 联合创始人 Mike Knoop 给出两条建议:一是“所有基准都是营销”,AI 实验室不会把对自己不利的基准放上模型卡;二是基准必须科学上有趣且足够难。团队据此把任务范围从 Zapier 专属场景扩展到更广的业务任务,发布时顶级模型得分低于 10%。

Body

AutomationBench 1.5 的关键改动是缩减任务数量,用省下的算力对每个任务运行多次试验,从而测量真实可靠性而非单次成功率。这种多试验设计让基准结果更能反映智能体在真实自动化工作流中的稳定表现。

对于正在做智能体评测的团队,这里有两个可复用的经验:难度校准到顶级模型低于 10% 能保证基准的区分度,而“每任务多次试验”的可靠性测量比单次通过率更接近生产环境的真实需求。

Original · Kaggle

When @SalimansRobin pitched building a business-agent benchmark at @zapier, co-founder @MikeKnoop gave him advice, including:

1. “All benchmarks are marketing”: AI labs won’t put your benchmark on a model card if they look bad on it. 2. “Make sure the benchmark is scientifically interesting, and make it hard”: Broadening beyond Zapier-specific tasks paid off. At launch, top models scored under 10%.

In his Bench Summit 2026 talk, Robin breaks down how AutomationBench 1.5 trims task counts to run multiple trials per task and measure true reliability.

Watch the full keynote: https://t.co/LquUB88XZk

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Industry" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include AutomationBench, Zapier, agents, benchmarks.

Source attribution: curated from Kaggle (@kaggle).