The Regression Tax: 技能如何同时帮助和损害LLM Agent

The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents

精选理由

这篇论文用大量实验告诉你,给Agent加技能可能适得其反。它揭示了三个具体原因,并指出减少回归比堆砌技能更重要,值得每个开发者读一读。

AI 摘要

该论文分析LLM Agent添加技能后的双面影响,基于近6,000次实验和两个办公自动化基准(如OfficeWorld和MiniWoB++),发现技能导致的回归(先前能完成的任务失败)频率高到足以使最佳技能的优势主要来自更少回归而非更多增益。论文识别三种回归原因:技能描述渗透(仅在上下文中存在即改变行为)、接地位移(技能流程覆盖输入解释)和验证位移(流程抑制输出检查)。分析持续失败发现现有技能过度强调程序指导(失败主因并非在此),而接地和验证支持不足。通过纠正评估伪影和轨迹分析,许多回归和持久失败可通过更好的接地和验证恢复。

原文 · arXiv cs.AI

The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents

Adding procedural skills to an LLM agent is typically evaluated by average improvement in task success. However, this metric hides an important cost: skills can also make agents worse. We measure both sides by comparing agents with and without skills across nearly 6,000 runs spanning two office automation benchmarks and three model harness stacks. This allows us to distinguish two outcomes. A regression is a task solved without skills but failed after skills are added. A residual failure is a task that fails both with and without skills. We find that regressions are substantial enough that the best performing skills outperform others primarily by regressing less, not by gaining more. We identify three causes of regression: (i) skill description osmosis, a skill changes an agent's behavior simply by being present in context, even when it is never invoked; (ii) grounding displacement, a skill's prescribed procedure overrides how the agent interprets its inputs; and (iii) verification displacement, where the procedure suppresses checks the agent would otherwise perform on its outputs. Analysing persistent failures reveals the same underlying pattern. Existing skills overemphasize procedural guidance the stage least often responsible for failure while under supporting grounding and verification, the dominant sources of remaining errors. After correcting evaluation artifacts and studying traces, we find many regressions and persistent failures recoverable through better grounding and verification. Procedural skills should be evaluated by decomposing their net effect into gains and regressions, not by aggregate improvement alone. We identify three regression modes skills should avoid, and find that reliability depends more on grounding and verification than on procedural skill choice.