论文精选

LayerRAG-Bench:面向智能体RAG的跨层可靠性基准

LayerRAG-Bench: A Cross-Layer Reliability Benchmark for Agentic Retrieval-Augmented Generation

精选理由

论文推出了LayerRAG-Bench,用8个领域240个任务测智能体RAG的可靠性陷阱。它告诉你schema规范化只解决一类问题,别指望一个补丁包治百病。

AI 摘要

LayerRAG-Bench覆盖8个企业领域、240个任务和9种故障场景,在来自OpenAI、Anthropic和Gemini的9个模型上生成38,880条任务级记录。schema规范化将schema漂移成功率从0.000提升到0.913,但无法修复过期证据、缺失工具输出、权限拒绝和错误会话上下文。仅基于groundedness的评估在过期证据和错误会话下产生大量假阳性。该基准支持按层评估原则:可靠性干预应只对其目标层生效,而非视为通用修复。

原文 · arXiv: Anthropic

LayerRAG-Bench: A Cross-Layer Reliability Benchmark for Agentic Retrieval-Augmented Generation

Agentic retrieval-augmented generation systems can produce answers that appear grounded while failing at the evidence, tool-contract, authorization, or session-state layer. We introduce LayerRAG-Bench, a controlled cross-layer reliability benchmark with 8 enterprise domains, 240 tasks, 9 fault scenarios, 2 contract modes, and 38,880 live task-level records across nine models from OpenAI, Anthropic, and Gemini. Schema normalization raises schema-drift success from 0.000 to 0.913, but stale evidence, missing tool output, denied permissions, and wrong-session context are not recovered by schema normalization. Groundedness-only evaluation also produces substantial false positives under stale and wrong-session evidence. These results support a layer-specific evaluation principle: a reliability intervention should be credited for repairing its target layer without being mistaken for a universal fix.