智能体评测一直是个难题，Agent Arena 用真实轨迹信号替代人类偏好，做智能体开发的团队可以直接参考这套方法论来改进自己的评测体系。
Agent Arena 团队指出，传统依赖人类偏好的评测方式无法扩展至智能体场景，因为人类难以判断30分钟轨迹中的数百次工具调用。他们构建了基于真实使用轨迹的客观信号评测方法，包括Bash错误、工具幻觉和“疯狂信号”等指标。这种方法能更准确地评估智能体在复杂任务中的表现，避免人类主观判断的局限性。该评测方法已应用于Agent Arena排行榜，为开发者提供更可靠的模型性能参考。
Agent Arena evals are fundamentally different. You can't ask humans to judge hundreds of tool call...
Agent Arena evals are fundamentally different. You can't ask humans to judge hundreds of tool calls across a 30-minute trace. So we built something different. We break down how the Agent Arena Leaderboard mines real usage traces for objective signals to move beyond human preference. 0:00 Human preference doesn't scale for agents 0:46 Mining traces for signals 1:51 Bash errors as objective signals 2:21 Tool hallucination 2:58 The insanity signal Your browser does not support the video tag. 🔗 View on Twitter 💬 2 🔄 1 ❤️ 6 👀 579 📊 3 ⚡