论文精选

Deep Research Agent 多轮评估:过程级反馈提升报告质量

Multi-Turn Evaluation of Deep Research Agents Under Process-Level Feedback

精选理由

做AI研究助手或智能体评估的团队,这篇论文揭示了当前DRA在多轮反馈下的真实瓶颈——单轮有效但多轮难持续,值得点开看具体实验设计和RGI方法。

AI 摘要

现有深度研究智能体(DRA)基准仅评估单次输出,忽略了反馈改进能力。研究者设计了多轮评估框架,对比自我反思和过程级反馈两种场景。过程级反馈通过研究缺口推断(RGI)方法,分析评分标准满足模式来指导改进。实验发现:自我反思几乎无净改进;单轮过程级反馈可提升8-15分,但多轮改进效果不叠加,因为重写时可能丢失已满足标准。这表明即使有针对性指导,DRA的可靠多轮改进仍具挑战。代码和结果已开源。

原文 · arXiv cs.LG

Multi-Turn Evaluation of Deep Research Agents Under Process-Level Feedback

Existing benchmarks for deep research agents (DRAs) assess only single-shot outputs, ignoring a key question: can DRAs improve their reports when guided by feedback? To investigate this, we conduct a multi-turn evaluation of DRAs under two feedback settings: self-reflection, in which the agent revises its report without any external diagnostic signal, and process-level feedback, in which the agent receives guidance targeting gaps in its research strategy. To enable process-level feedback, we design Research Gap Inference (RGI), a method that analyzes patterns of satisfied and unsatisfied rubric criteria to infer research-process gaps. Our analysis reveals three key findings: (i) under self-reflection, agents incorporate and regress on rubric criteria at nearly equal rates, yielding negligible net improvement; (ii) a single round of process-level feedback yields substantial gains, raising the normalized score by approximately $8$-$15$ points and yielding a roughly $35$-$40\%$ incorporation rate; (iii) these gains do not compound over subsequent turns, as agents regress on up to $24\%$ of previously satisfied criteria when rewriting the full report to address remaining gaps. Even with targeted guidance, reliable multi-turn improvement remains out of reach for the DRA architectures we evaluate. Our code and results are publicly available at https://github.com/sabharwalrishabh/Multi-Turn-Evaluation-of-DRAs.