Papers

HEAMS 框架发布:从安全、隐私、用户主权等维度全面评估智能体记忆系统

HEAMS: Holistic Evaluation of Agent Memory Systems

Why we picked it

做智能体记忆的团队该看看这个。MilaMem 这套开源基准把隐私、投毒、用户删除权都纳入评测,光跑性能分数会漏掉大问题,选型前建议先跑一遍。

一篇新论文提出 HEAMS(Holistic Evaluation of Agent Memory Systems),一个开源框架,主张把智能体记忆系统视为用户持久信息的保管者,而非只看任务表现。该框架将记忆系统的职责拆成五大类行为准则:性能、效率、安全(security)、隐私和用户主权(如可更正、可删除存储信息)。

Body

作者团队配套构建了覆盖这些维度的基准套件,开发了原型记忆系统 MilaMem,以及一个可包裹任意记忆系统的治理层(governance layer),用于控制变量地研究设计选择对行为的影响。实验对比多个记忆系统后发现:没有任何系统在所有性能基准上领先,性能得分高的系统可能在隐私、用户主权和安全维度表现很差。

治理策略的影响显著:召回策略能降低不当披露,但会牺牲帮助性;召回和保留策略都能提升抗记忆投毒能力。论文结论是仅靠性能与效率基准无法判断记忆系统是否尽到职责,代码已在 GitHub 开源。

Original · arXiv cs.AI

HEAMS: Holistic Evaluation of Agent Memory Systems

Memory systems allow AI agents to persist information across interactions, making them more personalized and more capable on long-horizon tasks. This persistence introduces new risks: stored information can be disclosed in inappropriate contexts, resist correction or deletion, be retained beyond what users desire, or be poisoned to influence later agent behaviour. Yet memory systems are typically evaluated on task performance and efficiency alone. We argue that a memory system should instead be understood as the custodian of its user's persistent information, with responsibilities that extend beyond helping an agent complete its tasks, and that it should be evaluated holistically against all such responsibilities. We present Holistic Evaluation of Agent Memory Systems (HEAMS), an open-source framework for evaluating agent memory systems in this way. First, we state the responsibilities of agent memory systems as explicit behavioural desiderata spanning performance, efficiency, security, privacy, and user sovereignty. Second, we assemble a benchmark suite that covers these desiderata and acknowledge the gaps that remain. Third, we develop MilaMem, a prototype memory system, and a governance layer that can wrap any memory system, enabling controlled study of memory system design choices. Fourth, we compare memory systems across the full HEAMS benchmark suite. We find that no memory system leads every performance benchmark, and strong performance scores may coexist with poor privacy, user-sovereignty, and security scores. Governance strongly influences results: recall policies can reduce inappropriate disclosure at a cost in helpfulness, while both recall and retention policies improve poisoning resistance. Performance and efficiency benchmarks alone therefore cannot reveal how well a memory system meets its responsibilities. Code: https://github.com/milatechtransfer/heams

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include agents, memory systems, AI safety, MilaMem.

Source attribution: curated from arXiv cs.AI (@Chirag Goel, Jacob Lavoie, Amirhossein Naghshzan, Jordan Bannister, Marzia Nouri, Mehar Bhatia, Matt Harris, Siva Reddy, Arsène Fansi Tchango, Gaétan Marceau Caron).