PapersSelected

Amazon 论文:环境变化后 Agent 技能库会失效,选择性修复可恢复价值

Why we picked it

做 Agent 记忆或技能库的都该看看:Amazon 实测过期技能能把 Opus 的 Terminal-Bench 2 通过率拖低 11.4 分,修复方案成本很低,只重放检查、不额外调用模型,适合开发者直接借鉴。

Amazon 的一项新研究发现,保存下来的 Agent 技能(skill)在环境变化后会反噬性能。技能库存储了历史经验,如文件路径、字段名和命令,本意是让 Agent 后续跳过重复工作;一旦应用或 API 发生变动,这些细节会静默过期。

Body

团队用 Claude Opus 4.7 和 Haiku 4.5 在 4 个设置上做了前后对比测试,其中包括 Terminal-Bench 2 和 IRS 税表任务。修复方法是不调用模型、直接重放保存过的检查项,只对冲突的技能内容做修复或重建。

数据对比很直观:在 Terminal-Bench 2 上,一个原本为 Opus 提升 4.7 分的技能,在环境变化后让通过率比不用技能还低 11.4 分;而维护机制把 Haiku 的税表字段准确率从带过期技能时的 30.5% 拉回到 97.5%。结论是复用技能前应先对照当前环境做校验,而不是等任务失败再排查。

Original · rohanpaul_ai

New Amazon paper finds that saved agent skills can hurt once the environment changes, and checking and repairing them selectively restores their value.

Agent skills store past experience, like file paths, field names, or commands, so agents can skip work later. When an app or API changes, some of that stored detail goes stale without warning.

They tested Claude Opus 4.7 and Haiku 4.5 on 4 settings, including Terminal-Bench 2 and IRS tax forms, before and after a change. Their fix replays saved checks witho1ut model calls and repairs or rebuilds only the conflicting skill content.

On Terminal-Bench 2, a skill that had added 4.7 points for Opus cut its pass rate 11.4 points below no skill after the change. Maintenance lifted Haiku's tax-form field accuracy from 30.5% with the stale skill to 97.5%.

Check saved agent skills against the current environment before reuse, instead of waiting for tasks to fail.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Claude, Agent, agents, AI safety.

Source attribution: curated from rohanpaul_ai (@rohanpaul_ai).