论文精选

用结构覆盖标准测试多智能体工作流

Testing Agentic Workflows with Structural Coverage Criteria

精选理由

多智能体系统测试长期依赖端到端指标,这篇论文给出了可落地的结构覆盖方法,做AI工作流测试的团队可以直接参考其DSPy实现来补全测试盲区。

AI 摘要

多智能体系统日益依赖显式工作流结构(如智能体、工具、访问规则和委托路径),但现有评估主要依赖端到端任务成功率或最终响应质量,难以验证这些声明结构是否真正被测试覆盖。该论文提出一种结构测试方法,将工作流表示为类型化协调图,推导覆盖义务(如可达智能体、允许/限制工具边、委托边),并利用DSPy生成可执行场景。在10个基准测试中,该方法成功覆盖了54/75的允许工具义务和36/48的委托义务,并发现了23/248的限制工具违规。结果表明,结构覆盖为多智能体工作流测试提供了有用的充分性层,能揭示声明结构是否被实际执行。

原文 · arXiv: OpenAI

Testing Agentic Workflows with Structural Coverage Criteria

Multi-agent systems increasingly expose explicit workflow structure: agents, tools, tool-access rules, restrictions, and delegation paths. Existing evaluations rely largely on end-to-end task success, benchmark scores, final-response quality, or prompt-level checks, which provide limited evidence that this declared coordination structure has actually been exercised. This makes it difficult to assess test-suite adequacy or detect structural regressions in tool access, restrictions, and inter-agent delegation. We address this gap with a structural testing approach for multi-agent workflow specifications. The approach represents each workflow as a typed coordination graph, derives coverage obligations over reachable agents, allowed tool edges, restricted tool edges, and delegation edges, and uses coverage-driven generation with DSPy-based scenario realization to produce executable tests. The graph fixes what must be covered; DSPy realizes those obligations as natural-language scenarios whose witnesses are checked at runtime. We implement the approach for OpenAI Agents SDK-style workflows and evaluate it on ten SDK-derived benchmarks comprising 49 reachable agents, 47 tools, and 403 structural obligations. Generated scenarios witness 54/75 allowed-tool obligations and 36/48 delegation obligations within a bounded refinement budget. The adversarial restricted-tool criterion elicits 23/248 restricted-call violations, separating workflows whose restrictions hold under probing from workflows with concrete misrouting failures. These results show that structural coverage provides a useful adequacy layer for multi-agent workflow testing: it does not replace semantic or end-to-end evaluation, but reveals whether declared agents, tool-access rules, restrictions, and delegation paths have been exercised.