arXiv 论文对比三种智能体评估框架：trace 评估 F1 达 0.90
Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents
做智能体评估的同学该看看这篇。它用同一批 trace 实测了三种评估框架：trace-based F1 到 0.90，纯文本方案误报多 5 倍，还有 24 个功能测试和安全红队零重叠的新失败。结论很实在：功能评估用 trace，安全验证用红队，两条腿走路。
arXiv 论文 2611.14493 对三种 LLM 智能体评估框架做了三阶段对比研究，覆盖基于 trace 的 LLM 评判、纯文本 LLM 评判加红队测试、确定性启发式规则三种范式，实验对象是两个企业分析智能体。Phase 1 通过重复测量量化评估框架的稳定性，97% 的配置达到稳定性阈值，且稳定性模式与具体智能体无关。Phase 2 在 72 个 ground-truth 测试用例上测量已知风险检测能力：trace-based 框架取得最佳精确率-召回率平衡（F1=0.90），纯文本评估召回率相当但误报多 5 倍。
Phase 3 评估未知风险发现，共挖出 24 个新失败案例，功能测试与安全红队测试之间零重叠，说明两者互补。论文还发现基于 LLM 的评估框架能发现人工审查遗漏的细微错误，评估框架可获得的信息量决定评估质量。
作者给出的落地建议是双框架策略：用 trace-based 评估做功能验证，用对抗性红队测试做安全验证。对需要在多工具工作流中部署分析智能体的企业工程团队，这是选型评估方案的直接参考。
Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents
Deploying LLM-based analytics agents in enterprise settings requires evaluation frameworks that can reliably detect failures across complex, multi-tool workflows. We present a three-phase comparative study of three evaluation frameworks, each representing a distinct evaluation paradigm (trace-based LLM judging, text-only LLM judging with red-teaming, and deterministic heuristics), applied to two analytics agents in a controlled research setting using frozen execution traces. Phase 1 quantifies evaluation harness stability through repeated measurements, finding that 97% of configurations meet the stability threshold and that stability patterns are agent-independent. Phase 2 measures known risk detection on 72 ground-truth test cases, where the trace-based framework achieves the best precision-recall balance (F1=0.90) while text-only evaluation matches on recall but generates 5x more false alarms. Phase 3 evaluates unknown risk discovery, uncovering 24 novel failures with zero overlap between functional testing and security red-teaming. Our results demonstrate that (1) the information available to the evaluation harness determines evaluation quality, (2) functional and security evaluation are complementary, and (3) LLM-based evaluation harnesses surface subtle errors that human reviewers miss. We recommend a two-framework strategy: trace-based evaluation for functional assessment and adversarial red-teaming for security validation.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include 智能体评估, trace-based evaluation, red-teaming, LLM judging.
Source attribution: curated from arXiv cs.AI (@Howard Zhang, Chuck Arvin, Logan Brown, Mihir Bhatia, Neelu Choudhary, Lily Miao).
