Kaggle 举办 Bench Summit，要给 AI 基准测试立行业标准
Kaggle 办了个基准测试峰会，把现在榜单分数为啥不可信讲得很透，跑评测的都该看看。
Kaggle 的 Bench Summit 正式开幕，Janie Gu 在开场演讲中指出现有基准测试的三大问题：排行榜与生产环境中 Agent 的实际表现脱节、评测随版本迭代快速过时且计算成本高、各家自建评测框架导致分数差异来自基础设施而非模型质量。她提出 Kaggle 的解决路径：以真实工作流为基础构建基准、用标准化基础设施保证分数可复现、由人类和 AI 评测专家背书方法论。Kaggle 希望在基准测试领域复制其在竞赛领域的行业地位。
Bench Summit by Kaggle has officially kicked-off! 🎤
The path forward starts here. We started strong with @janiejgu sharing key insights on the problems with benchmarks today and Kaggle’s mission to build trust and become the industry standard for benchmarks, the way we are for competitions.
During the session, she shared three ways benchmarks fail today: 1. Relevance: Public leaderboards may not resemble what your agents do in production. 2. Compute: Evals rot within a release cycle, so staying fresh is too expensive without ongoing compute. 3. Verification: Everyone runs on their own harness, meaning scores vary wildly due to infrastructure, not model quality.
To get numbers you can actually trust, Janie highlighted three essentials: 1. Real workflows as the foundation for benchmarks 2. Accurate and reproducible scores via standardized infrastructure 3. Rigorous methodology backed by human and AI evaluation experts
Here’s a sneak peak of the early moments:
