评估编程智能体工作记忆

Measure Before You Manage: Evaluating Agent Working Memory in Coding Agents

精选理由

这篇论文揭示了编程智能体工作记忆的语义异质性,对优化AI编程助手内存管理有实用价值。

AI 摘要

研究人员分析了55个编程智能体轨迹,发现语义不同的工作记忆对象表现出不同的保留和压缩行为。研究测试了两种基于语义的记忆管理策略:对象感知压缩策略和基于检索的策略。研究显示校准收益可能不会转移到保留任务中,且相同的令牌预算并不意味着相同的有效上下文或管理成本。

原文 · arXiv cs.AI

Measure Before You Manage: Evaluating Agent Working Memory in Coding Agents

Agent working memory is heterogeneous. Objects such as instructions, artifacts, tool outputs, and agent-generated state play different semantic roles and exhibit different size, retention, and representation profiles. Recent work has begun to explore memory-management mechanisms that account for such heterogeneity. This work focuses on semantic heterogeneity and studies how it should shape the management and evaluation of working memory in coding agents. Across 55 archived coding-agent trajectories, we find that semantically different working-memory objects exhibit distinct retention and compression behavior. This heterogeneity motivates semantically informed memory management. We study two semantically informed strategies: an object-aware compression policy and a retrieval-based policy. Their evaluation shows that calibration gains may not transfer to held-out tasks, and that equal token budgets do not imply equal delivered context or management cost. A real-system replay further exposes serving limits that nominal budgets alone do not capture. Together, these results show why semantic structure matters for agent working memory and why evaluating memory-management strategies requires more than a nominal token budget. We organize these lessons into four levels: stored state, delivered context, management work, and task or process outcome.