论文精选72°

LogDx-CI:首个日志缩减工具基准,助LLM根因诊断

LogDx-CI: Benchmarking Log Reduction Tools for LLM Root-Cause Diagnosis

精选理由

做CI调试或AI编程智能体的团队终于有了可依赖的日志缩减基准——LogDx-CI直接告诉你哪种工具省钱又有效,建议做DevOps或Agent开发的点开看。

AI 摘要

CI失败日志通常庞大且嘈杂(中位数5000行,最多20万行),编程智能体依赖上游工具将日志缩减为可管理的上下文,但此前缺乏公开的实证比较。LogDx-CI基准测试比较了11种上下文缩减工具(包括原始、tail、grep、三种RTK模式、两种LLM map-reduce摘要器、三种混合路由器)在35个真实GitHub Actions失败案例上的表现,由3个LLM调试器家族评分。关键发现包括:混合grep+tail路由器在成本和质量上占据帕累托前沿,前两名方法每个案例约0.03美元,质量与独立grep相当但token减少4.5倍;在智能体循环中,不同缩减工具的质量差距缩小了7倍,但弱上下文会导致智能体发出2-4倍的工具调用;跨家族LLM摘要-调试器组合(gpt-5-mini摘要器+Claude Haiku调试器)优于同家族组合,平均提升0.071分。所有数据、代码和可复现基础设施均已公开。

原文 · arXiv: OpenAI

LogDx-CI: Benchmarking Log Reduction Tools for LLM Root-Cause Diagnosis

CI failure logs are large (median 5k lines, max 200k in this corpus) and noisy. Coding agents that try to debug them depend on an upstream tool to reduce the log to a manageable context, but the field has had no public empirical comparison of which reductions preserve enough evidence for downstream LLM diagnosis. We introduce LogDx-CI, a benchmark that compares 11 context-reduction tools (raw, tail, grep, three RTK modes, two real LLM map-reduce summarizers, three hybrid routers) on 35 real GitHub Actions failure cases, scored by 3 LLM debugger families (Claude Haiku 4.5, Claude Sonnet 4.6, OpenAI gpt-5-mini) plus a Sonnet 4.6 tool-using agent. We report three load-bearing findings. (1)~Hybrid grep+tail routers dominate the cost-quality Pareto frontier; the top two methods score 0.670 / 0.666 at $\sim$ \$0.03 per case, same-ballpark quality as standalone grep at $4.5\times$ fewer tokens. (2)~In the agent-loop regime, the quality range across reduction tools collapses $7\times$ (single-shot spread 0.42 $\to$ agent-loop spread 0.059); the agent rescues weak contexts via follow-up tool calls. However, cost differences persist: weak contexts force the agent to issue 2--4$\times$ more tool calls to recover. (3)~A cross-family LLM-summary pair (gpt-5-mini summarizer feeding a Claude Haiku debugger) beats the same-family pair by $+0.071$ averaged across four diagnoser variants, falsifying the self-call-bias hypothesis on this task. The gpt-5-mini summarizer is also the agent-loop \#1 method (score 0.749) at $0.37$ tool-calls per case and $10\times$ lower reducer cost than the Haiku summarizer (\$0.18 vs \$1.75 per case). All data, code, per-case bundles, and reproducibility infrastructure are public.