PapersSelected

AutoRAGBench:编码智能体在 RAG 开发任务上超过已发表的人类最佳方案

Why we picked it

这篇论文用 9 个 RAG 任务测试 GPT-6.1 Sol + Codex CLI,agent 从零开始拿到 56.1 分,超过了已发表人类最佳方案的 44.4 分。做 RAG 系统的工程师应该看看,里面还比较了 9 种 agent 配置的差异。

AutoRAGBench 是一项新的基准测试,让编码智能体独立完成 9 个 RAG 开发任务,无起始代码、无提示方案,仅有 6 小时时间和 1 块 H800 GPU,需自行分析数据、搭建管线并根据实验迭代。

Body

测试结果:GPT-6.1 Sol 搭配 Codex CLI 平均得分 56.1,而由每个任务上已发表最佳人类方法组成的复合基线平均为 44.4。若智能体从人类设计的系统起步而非从零构建,可在 7 个任务上平均再加 16.6 分。

论文同时比较了 9 种智能体配置,并分析了各配置在 6 小时内开发过程的演变方式,说明智能体不只靠初始设计,更多依赖实验驱动的持续改进。对做 RAG 的工程师来说,这提示 agent 自动探索管线设计可能成为一条可行路线。

Original · elvis

Can coding agents build better RAG systems than the best published human designs? AutoRAGBench gives an agent nine RAG development tasks to measure this. There is no starter code and no suggested approach. The agent gets six hours and one H800 GPU to analyze the data, build a pipeline, and refine it from experiments. GPT-6.1 Sol with Codex CLI averages 56.1. The composite human baseline, which takes the best published method on each task, averages 44.4. When the agent starts from a human-designed system instead of from scratch, it adds 16.6 points on average across seven tasks. The paper compares nine agent setups and also studies how each one's development process unfolds over the six hours. Paper: arxiv.org/abs/2611.25582 Chat with Paper: academy.dair.ai/papers/autorag…📊 20 ⚡

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include AutoRAGBench, RAG, Codex CLI, agents.

Source attribution: curated from elvis (@omarsar0).