DRFLOW 基准:评估智能体预测个性化工作流的能力

DRFLOW: A Deep Research Benchmark for Personalized Workflow Prediction

精选理由

想测测你家智能体能不能给出真正可执行的步骤?DRFLOW 用 100 个企业任务逼真考核,DRFA 也才比普通基线高 10%,挑战不小。

AI 摘要

DRFLOW 是一个用于评估智能体从异构来源预测个性化工作流的新型基准,包含 100 个任务、5 个领域、1,246 个参考步骤,并基于 3,900 多个来源构建。它定义了 7 项诊断指标,涵盖事实基础、步骤恢复、结构排序、条件解析和个性化。作者还提出了 DRFLOW-Agent (DRFA) 作为参考代理,其在平均 F1 分数上比强基线代理提升最多 10.02%,但仍有较大改进空间。

原文 · arXiv cs.AI

DRFLOW: A Deep Research Benchmark for Personalized Workflow Prediction

Deep research (DR) systems are increasingly used for complex information-seeking tasks, but existing works mainly focus on generating reports and summaries. In contrast, many enterprise tasks instead require an agent to identify concrete workflows which is a sequence of action-steps. For example, rather than summarizing budgeting policies, an agent should be able to determine the steps needed to answer a question such as: "How do I request new headcount given a fixed budget?". Therefore, we introduce DRFLOW, a benchmark for evaluating personalized workflows predicted by agents from heterogeneous sources. Each task requires the agent to identify relevant evidence from scattered sources, then use that evidence to predict the correct action-step sequence for the user's task. DRFLOW contains 100 tasks across five domains, with 1,246 reference workflow steps grounded in more than 3,900 sources. We define seven diagnostic metrics covering factual grounding, step recovery, structural ordering, condition resolution, and personalization. We further present DRFLOW-Agent (DRFA), a workflow-oriented reference agent to predict personalized workflow. We show that although DRFA improves over strong baseline agents (upto 10.02% average F1 score), there is substantial room for improvement remains across these workflow metrics, indicating that predicting complete and correct personalized workflows remains a challenging frontier for deep research.