ALE 揭示了当前 AI 智能体在真实专业任务上的真实水平，做智能体开发或评估的团队值得关注——它可能是衡量 Agent 能力的最后一把尺子。
Agents' Last Exam (ALE) 是一个针对 AI 智能体的新基准测试，包含来自 55 个行业的 1000 多个真实专业任务，所有任务都源自实际专家工作，而非合成数据。测试结果显示，最佳智能体在最简单任务上得分低于 50%，在困难任务上低于 10%，最前沿模型在最高难度任务上通过率为 0%。模型选择对性能的影响大于工具链（harness），且增加 token 消耗并不能提升结果。智能体常见失败模式包括策略错误（47%）、领域知识缺失（31%）和执行错误（22%），且 34% 的任务需要 GUI 软件，但智能体倾向于回避并采用 CLI 变通方案。
The last benchmark for agents? Agents' Last Exam (ALE) evaluates agents on 1,000+ real world profess...
The last benchmark for agents? Agents' Last Exam (ALE) evaluates agents on 1,000+ real world professional tasks across 55 industries, all sourced from actual expert work. Not synthetic. Not multiple choice. Real deliverables, graded deterministically. Key findings: - Best agents score <50% on the easiest tier, <10% on the hardest - 82% on Terminal-Bench drops to 23% on ALE-CLI eval with the same setup - Hardest tier: most frontier agents hit 0% pass rate - Spending more tokens doesn't improve results - Each run tracks harness, model, pass rate, token usage, and cost Harness vs. model: - Best harness scores 24.0%, worst scores 19.1% (same model). That's a 4.9pp gap. - Model choice drives more performance variation than the harness. - Most efficient setup used 160M tokens for 39.6%. Least efficient burned 1,373M tokens for 40.5%. Where agents break (Agents often say "Done. All checks pass." while the output is wrong) - 47% of failures: wrong strategy or gave up early - 31%: missing domain knowledge - 22%: execution bugs and format errors - 34% of tasks need GUI software, agents avoid it and hack CLI workarounds Very excited to see a benchmark like this. Big kudos to everyone who contributed. 💬 1 🔄 1 ❤️ 0 👀 88 📊 1 ⚡