AITOPAI热报
PapersSelected

Continual-ARC 基准测试：Agent 记忆系统难以泛化，训练 9B 模型权重反而有效

elvis@omarsar0
Post
View the post
Why we picked it

如果你在做 Agent 记忆相关的工作，这篇很对味。作者拿 Continual-ARC 实测了 OpenClaw、Hermes Agent 和 Prime Agent，结论是都不如直接训练一个 9B 小模型。对纠结外挂记忆还是训权重的人有直接参考价值。

Unify.ai 团队发布 Continual-ARC 基准测试，评估 Agent 能否从经验中持续学习。结果显示 OpenClaw、Hermes Agent 和 Prime Agent 三个主流 Agent 框架都没有从经验中有效学习：既记不住有价值的信息，也无法泛化到新任务。对照组中，训练一个 9B 小模型的权重则同时做到了记忆和泛化。

Body

论文提出的核心思路是让 harness 与模型共同进化（co-evolving）。当前 harness 中的记忆机制只适合简单、一次性、通用任务，遇到新颖且领域特定的任务就难以泛化。研究表明 harness 的角色正在变化：它不只是给模型提供工具、上下文和引导，还可以成为训练流水线的一部分，用来解锁模型的持续学习能力。

对开发者而言，这项分析意味着依赖外部记忆组件的 Agent 方案存在天花板，而把经验编码进模型权重可能是更可行的方向。harness 本身仍重要，但可以保持精简高效。

Original · elvis

Highly recommended read. This is a topic I have been researching and writing about recently. The way we use agent memory in harnesses today works, but mostly for simple, one-off, and general tasks. For more novel, complex, and domain-specific tasks, you will often find that current harness and memory systems don't generalize so well. So the big question is: how can we encode continual learning capabilities in models and harnesses? This is where the idea of co-evolving harnesses and models begins to make sense. This work, along with the benchmark (Continual-ARC), provides an interesting analysis of this. The harness is not just for steering the model and providing tools and context; it is increasingly becoming a key part of training pipelines to unlock continual learning in models. The harness continues to be important, but it can remain minimal and efficient. Daniel Lenton @DanielLenton1 OpenClaw, Hermes Agent and Prime Agent don't meaningfully learn from experience. On Continual-ARC, our new benchmark at @letsunifyai , none of them were able to memorize anything of value and none were able to generalize to new tasks. Training a small 9B model's weights did both.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Continual-ARC, Agent, 智能体, 持续学习.

Source attribution: curated from elvis (@omarsar0).

#Continual-ARC#Agent#智能体#持续学习#基准测试

Keep reading

Recent stories with the same tag or section
PapersBuildersMulti-source13:56
Chinese contentContinual-ARC：用 ARC-AGI 三代任务流测试智能体的终身学习能力

这个基准把记忆型和权重型学习放进同一条 ARC-AGI 任务流里打分，结论挺扎心：单纯加记忆的 agent harness 还不如缓存演示的笨基线，泛化还是得靠改权重。做长期记忆和智能体框架的人应该看看，它给了可复现的评测管线。

#Continual-ARC#ARC-AGI#Qwen3.5#GLM
Event
arXiv cs.LG@Daniel LentonSource
PapersUS vs ChinaMulti-sourceSelected05:23
OpenAI research finds chain-of-thought monitorability declines as models scale

The non-obvious takeaway is that safety monitoring based on reading a model's reasoning traces may silently weaken as capability grows, meaning audits that work today may not work on next year's models. Safety researchers, alignment teams and enterprises using CoT inspection for oversight should benchmark monitorability on their own fine-tunes. The training-lifecycle framing gives practitioners a concrete methodology for measuring when and where this degradation starts.

#OpenAI#chain-of-thought#AI safety#reasoning models
Event
elvis@omarsar0Source
PapersBuildersSelected03:31
Perplexity red-teams SPACE sandbox with 12 AI models

The counterintuitive result — general-purpose models beating cyber-specialized ones at sandbox escapes — suggests teams should benchmark attackers on their own infrastructure rather than trusting security-tuned model variants. Builders of computer-use agents and sandbox environments can apply the prompting guidance and DNS spoofing finding directly when hardening isolation boundaries. Open-source access to SPACE makes it a practical starting point for running comparable red-team evaluations.

#Perplexity#SPACE#AI safety#agents
Aravind Srinivas@AravSrinivasSource
PapersBuildersOfficialSelected02:40
Apple Details Weighted State Souping for Long-Context SSM Retrieval

The non-obvious point is that cached SSM states can be reused like precomputed retrieval index entries, so combining them with learned weights changes how memory-heavy applications manage context cost. Builders of agents, RAG pipelines and personalized assistants can evaluate this as a cheaper alternative to re-encoding long contexts on every query. Compared with transformer-based long-context methods, it trades architectural familiarity for fixed memory and faster inference.

#SSM#state space models#RAG#Apple
Primary sourceApple ML ResearchSource
PapersBuildersOfficialSelected02:40
Apple Research Links Gating to In-Context Learning in State Space Models

The non-obvious takeaway is that SSM weaknesses on retrieval may be fixable through gating design rather than requiring a wholesale return to Transformers. Model architects and ML engineers evaluating efficient alternatives to attention should study the paper's analysis to guide architecture choices. Compared with prior empirical comparisons between SSMs and Transformers, this work offers a mechanistic explanation rather than just benchmark observations, which is useful for anyone building long-context inference systems.

#SSM#gating#in-context learning#Transformers
Primary sourceApple ML ResearchSource