用 LLM 提取轨迹摘要降低智能体评测成本
Task-Efficient Agent Evaluation Using Cached Traces
评测智能体跑一遍基准太烧钱？这篇论文用 LLM 摘要轨迹来预测分数，SWE-bench 上误差从 0.13 降到 0.08，只跑一个任务就能估分。
一篇 arXiv 论文提出用 LLM 从智能体执行轨迹中抽取结构化摘要，再用标准文本编码器嵌入，构建参考系统的低维表征来预测新系统的基准分数。在 SWE-bench Verified 上，将几何预测结合 item-response 模型后，仅执行 1 个随机任务时平均绝对误差从 0.13 降到 0.08（Wilcoxon p < 10^-4）；在 Terminal-Bench 2.0 上从 0.14 降到 0.12。该方法还能抑制原始轨迹嵌入中的系统身份、模型家族和 harness 信号，在给定任意执行任务数下均优于基线。
Task-Efficient Agent Evaluation Using Cached Traces
Efficient benchmarking estimates a system's full benchmark performance from a small number of task executions. Standard approaches rely solely on the binary correctness of executions from a collection of already-evaluated reference systems. An agentic system, however, produces a rich trace for each execution that can reveal substantially more about its capabilities. Leveraging this information requires useful trace representations. We propose representing a trace by extracting a structured summary of it with an LLM and embedding the summary with a standard text encoder. These representations preserve correctness- and task-related signal while minimizing the presence of system identity-, model family- and harness-related signals that dominate raw trace embeddings. Given this representation for each task execution from all reference systems, we construct low-dimensional representations of the systems and predict a new system's score from its position among the reference systems. On SWE-bench Verified and Terminal-Bench 2.0, augmenting an item-response model with this geometry-based prediction reduces mean absolute error from 0.13 to 0.08 (Wilcoxon $p < 10^{-4}$) and from 0.14 to 0.12 (Wilcoxon $p < 10^{-3}$), respectively, when provided a single random executed task. Further, given access to the full set of reference systems, our approach dominates baselines for any given number of executed tasks on both benchmarks.
